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Analyticity, Convergence and Convergence 
Rate of Recursive Maximum Likelihood 
Estimation in Hidden Markov Models 

Vladislav B. Tadic^ 
Abstract 

This paper considers the asymptotic behavior of the recursive maximum likelihood estimation in hidden Markov 
models. The paper is focused on the analytic properties of the asymptotic log-likelihood and on the point-convergence 
and convergence rate of the recursive maximum likelihood estimator Using the principle of analytical continuation, the 
analyticity of the asymptotic log-likelihood is shown for analytically parameterized hidden Markov models. Relying 
on this fact and some results from differential geometry (Lojasiewicz inequality), the almost sure point-convergence 
of the recursive maximum likelihood algorithm is demonstrated, and relatively tight bounds on the convergence rate 
are derived. As opposed to the existing result on the asymptotic behavior of maximum likelihood estimation in hidden 
Markov models, the results of this paper are obtained without assuming that the log-likelihood function has an isolated 
maximum at which the Hessian is strictly negative definite. 

Index Terms 

Hidden Markov models, maximum likelihood estimation, recursive identification, analyticity, Lojasiewicz inequal- 
ity, point-convergence, convergence rate. 

I. Introduction 

Hidden Markov models are a broad class of stochastic processes capable of modeling complex coiTelated data and 
large-scale dynamical systems. These processes consist of two components: states and observations. The states are 
unobservable and form a Markov chain. The observations are independent conditionally on the states and provide 
only available information about the state dynamics. Hidden Markov models have been formulated in the seminal 
paper yj, and over last few decades, they have found a wide range of applications in diverse areas such as acoustics 
and signal processing, image analysis and computer vision, automatic control and robotics, economics and finance, 
computation biology and bioinformatics. Due to their practical relevance, these models have extensively been studied 
in a large number of papers and books (see e.g., lH, IIT2I and references cited therein). 
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Besides the estimation of states given available observations (also known as filtering), the identification of model 
parameters are probably the most important problem associated with hidden Markov models. This problem can be 
described as the estimation (or approximation) of the state transition probabilities and the observations conditional 
distributions given available observations. The identification of hidden Markov models have been considered in 
numerous papers and several methods and algorithms have been developed (see [8 Part II], lfT2l and references 
cited therein). Among them, the methods based on the maximum likelihood principle are probably one of the 
most important and popular. Their various asymptotic properties (asymptotic consistency, asymptotic normality, 
convergence rate) have been analyzed in a number of papers (see [|T], ||5|, HOl, lUIl, GH - ED, |l24l, ||28ll . 
ED, ll34l : see also lH Chapter 12], lfT2l and references cited therein). Although the existing results provide an 
excellent insight into the asymptotic behavior of maximum likelihood estimators for hidden Markov models, they all 
crucially rely on the assumption that the log-likelihood function has a strong maximum, i.e., an isolated maximum 
at which the Hessian is strictly negative definite. As the log-likelihood function admits no close-form expression 
and is fairly complex even for small-size hidden Markov models (four or more states), it is hard (if not impossible 
at all) to show the existence of an isolated maximum, let alone checking the definiteness of the Hessian. 

The differentiability, analyticity and other analytic properties of functionals of hidden Markov models similar 
to the asymptotic likelihood (mainly entropy rate) have recently been studied in lfT3l . llT4ll . lITSl . ||29l . If30l . Il35l . 
Although very insightful and useful, the results presented in these papers cover only models with discrete state and 
observation spaces and do not consider the asymptotic behavior of the maximum likelihood estimation method. 

In this paper, we study the asymptotic behavior of the recursive maximum likelihood estimation in hidden Markov 
models with a discrete state-space and continuous observations. We establish a link between the analyticity of the 
asymptotic log-likelihood on one side, and the point-convergence and convergence rate of the recursive maximum 
likelihood algorithm, on the other side. More specifically, relying on the principle of analytical continuation, we 
show under mild conditions that the asymptotic log-likelihood function is analytical in the model parameters if 
the state transition probabilities and the observation conditional distributions are analytically parameterized. Using 
this fact and some results from differential geometry (Lojasiewicz inequality), we demonstrate that the recursive 
maximum likelihood algorithm for hidden Markov models is almost surely point-convergent (i.e., it has a single 
accumulation point w.p.l). We also derive tight bounds on the almost sure convergence rate. As opposed to all 
existing results on the asymptotic behavior of maximum likelihood estimation in hidden Markov models, the results 
of this paper are obtained without assuming that the log-likelihood function has an isolated strong maximum. 

The paper is organized as follows. In Section [III the hidden Markov models and the corresponding recursive 
maximum likelihood algorithms are defined. The main results are presented in Section [III too. Section Hill provides 
several practically relevant examples of the main results. Section |IV] contains the proofs of the main results, while 
the results of Section |lll] are shown in Section |V] 
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II. Main Results 

In order to state the problems of recursive identification and maximum likelihood estimation in hidden Markov 
models with finite state-spaces and continuous observations, we use the following notation. iV^; > 1 is an integer, 
while X = {1, . . . , Nx}- dj, > 1 is also an integer, while 3^ is a Borel-measurable set from M"*". {p{x'\x)}x,x'ex 
are non-negative real numbers such that J2x'ex -P(^'N) — 1 for ^^ch x ^ X. {Q{-\xy}xex are probability measures 
on y. {(^n, i^n)}ri>o is an A" X ^-valued stochastic process which is defined on a (canonical) probabihty space 
(ri,J^, P) and satisfies 

P(r„+i € B; Xn+1 = x\Xo, Yo,..., X„, r„) = QiB\x)p{x\Xn) 

w.p. 1 for all X £ X, n > 0, and any Borel measurable set B from y. On the other side, dg is a positive integer, while 
is an open set from M'*''. {pe{x'\x)}x^x'ex are Borel-measurable functions of e 6 such that pg{x'\x) > and 
J2x"<£xPsi^"\'^) = 1 foi" 9 £ Q, x,x' £ X. {qeiy\x)}xex are Borel-measurable functions of {6,y) £ Q x y 
such that qeiy\x) > and Jyqg{y'\x)dy' ^ 1 for all 6 e O, x e X, y e y. For 9 e Q, {{X^,Y^)}n>o is an 
X X 3^-valued stochastic process which is defined on a (canonical) probability space {n,!F,Pg) and admits 

Pe{Yf,+,eB,X^,^,^x\X'oX,---,K,Yn)^ f qe{y\x)pe{x\X^,)dy 

Jb 

w.p.l for each x ^ X, n > 0, and any Borel measurable set B from Y. Finally, /(•) stands for the asymptotic 
value of the log-likelihood function associated with data {l"„}„>o. It is defined by 

f{9) = lim E(-logp^{Y,,...,Y,,)] 

for 6' e 6, where 

n 

Pe(m^---^yn) = X! Pe{Xo = xo)'[[{q0{yk\xk)pe{xk\xk-i)) 

Xo,....XnG^ i — 1 

for 9ee,yi,...,yney,n>0. 

In the statistics and engineering literature, {(X,i, l^)}„>o (as well as {(X^, y^)}„>o) is commonly referred to 
as a hidden Markov model with a finite state-space and continuous observations, while X„ and Yn are considered 
as the (unobservable) state and (observable) output at discrete-time n. On the other hand, the identification of 
{{Xn,Yn)}n>o is regarded to as the estimation (or approximation) of {p{x'\x)}x.x'ex and {Q{-\x)}xex.yey 
given the output sequence {y„}„>o- If the identification is based on the maximum likelihood principle and 
the parameterized model {pe{x'\x)}x,x'ex, {qe{y\x)}xex,yey^ the estimation reduces to the maximization of 
the likelihood function /(•) over 6. In that context, {(X^, yjf )}„>o is considered as a candidate model of 
{{Xn, Yn)}n>o- For morc details on hidden Markov models and their identification see ||8] Part II] and references 
cited therein. 

Since the asymptotic mean of \ogpg{Yi, . . . , Yn)/n is rarely available analytically, /(•) is usually maximized by 
a stochastic gradient algorithm, which itself is a special case of stochastic approximation (for details see ifTSl . 
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and references cited therein). To define such an algorithm, we introduce some further notation. For E R 

x,x' e X, y e y, let 

rg{y\x',x) ^ qe{y\x')pe{x'\x), 

while Re{y) is an matrix whose entry is rg{y\i,j) (i.e., Re[y) = Ve{y\h j)\i.j£x)- On the other 

side, for 9 G W^" , u e [0, oo)^- \ {0}, V G R^exN^^ y ey, 1 < k < dg, let 

(l>e{u,y) = \og{e^ Re{y)u), 

Fg{u,V,y) :^Wg4>e(u,y) + VVu<l)e{u,y), 
R0{y)u 

Gg[u,y) - 



e^Re{y)u 

Hg{u,V,y) = VgGg{u,y) + VVuGeiu,y) 
where e = [1 . . . 1]^ G R^^. With this notation, a stochastic gradient algorithm for maximizing /(•) can be defined 



as 



9,1+1 = + anFg,^{Un,Vn,Yn+i), (1) 
Un+1 = Ge„ + i{Un,Yn+i), (2) 
Vn+1= Hg^^,{Un,Vn,Y„+i), n>0. (3) 

In this recursion, {a„}„>o denotes a sequence of positive reals. Oq E M'^", Uq E and Vq E R'^d'^n^ ^j.g j-^^dom 
variables which are defined on the probability space {il,!F,P) and independent of {Yn}n>o- 

In the literature on hidden Markov models and system identification, recursion ([T]l - OJ is known as the recursive 
maximum likelihood algorithm, while subrecursions (|2]i and ([3]) are referred to as the optimal filter and the optimal 
filter derivatives, respectively (see fSl for further details). Recursion ([T]l - S usually includes a projection (or 
truncation) device which prevents estimates {6'„}„>o from leaving Q (see |25| for further details). However, in 
order to avoid unnecessary technical details and to keep the exposition as simple as possible, this aspect of algorithm 
(H) - (O is not considered here. Instead, similarly as in 1251 . our results on the asymptotic behavior of algorithm 
([T]i - © (Theorems |2] and [3]) are expressed in a local form. 

Throughout the paper, unless stated otherwise, the following notation is used. For an integer d > 1, denotes 
the set of d-dimensional probability vectors (i.e., T"^ = {u E [0, oo)'' : e'^u = 1}), while C'' and C'*'^'' are the sets 
of c?-dimensional complex vectors and d x d complex matrices (respectively). || • || is the Euclidean norm in R'^ or 
C^, while d{-, •) is the distance induced by this norm. For a real number S E (0, oo) and a set ^ C C*, Vs{A) is 
the (complex) (5-vicinity of A induced by distance d{-, •), i.e., 

Vs(A) = {w e : d{w,A) < S}. 

S is the set of stationary points of /(•), i.e., 

= {61 e e : V/(6l) = 0}. 
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Algorithm ([T]l - (|3]l is analyzed under the following assumptions. 

Assumption 1: lim„^oo ctn = 0, limsup^^^^ Ic^rT+i ~ "^n ^1 °° Sj^o ~ Moreover, there exists a 
real number r e (1, oo) such that X)^o '^n7n'^ ^ 
Assumption 2: {X„}„>o is geometrically ergodic. 

Assumption 3: There exists a function se(2/|a;) mapping {6, x,y) G Q x X x y into [0, cx)), and for any compact 
set Q C &, there exists a real number eq E (0, 1) such that 

£QSe{y\x') < rg{y\x',x) < Sg'^sgiylx') 

for all 9eQ, x,x' eX,yey. 

Assumption 4: For each y £ X (l)g{u,y) and Gg{u,y) are real-analytic functions of (6',m) on entire 8 x V^"'. 
Moreover, (t)g{u,y) and Gg{u,y) have (complex-valued) analytical continuations y) and Gjj{w,y) (respec- 

tively) with the following properties: 

i) (j)ri{w,y) and Gj^{w,y) map {T],w,y) £ C^" x x y into C and (respectively). 

ii) (t>e{u,y) = (t>e{u,y) and Gg{u,y) = Gg{u,y) for all 6* e 6, it G y G 3^. 

iii) For any compact set Q C 0, there exist real numbers 5q E (0, 1), A'q £ [1, oo) and a Borel-measurable 
function ^Ijq : y ^ [l,oo) such that (f)r^{w,y) and Gn{w,y) are analytical in (?7, w) on V5q(Q) x Vsq{V^^) 
for each y G 3^, and such that 

\(j)n{w,y)\ < i}Q{y), 

\\G,{w,y)\\ < Kq, 

J ^l{y')Q{dy'\x) <^ 

for all 77 G Vsq{Q), w G Vs,,{V^-), xeX^yey. 

Assumption [T] corresponds to the properties of step-size sequence {q!„}„>o and is commonly used in the 
asymptotic analysis of stochastic approximation algorithms. It holds if — l/n"- for n>\, where a G (3/4, 1]. 

Assumptions 12] and |3] are related to the stability of the model {{Xn,Yn)}n>o and its optimal filter In this or 
similar form, they are involved in the analysis of various aspects of optimal filtering and parameter estimation in 
hidden Markov models (see e.g., Q, lISl, EO), Ol, 113 - EH, El, ED, |I33], 01, Eg); see also M Part II] 
and references cited therein). 

Assumption H corresponds to the parametrization of candidate models {(X^^, y^)}„>o. Basically, Assumption H 
requires transition probabilities pg{x'\x) and observation conditional densities qo{y\x) to be analytic in 9. It also 
requires pg{x'\x) and qg{y\x) can be analytically continuable to a complex domain such that the corresponding 
continuation of the optimal filter transfer function Ge{u,y) is analytic and uniformly bounded in {6,u). Although 
these requirements are restrictive, they still hold in many practically relevant cases and situations. Several examples 
are provided in the next section. 



6 



In order to state our main results we rely on the following notation. {7,i}„>o is a sequence of real numbers 
defined by 70 = 1 and 

Tl-l 

1=0 

for n > 1. Event A is defined as 

A = jsup||6'„|| < 00, inf d{0n,de) > ol . 

Ln>0 "■>0 J 

With this notation, our main results on the properties of objective function /(•) and algorithm ([B - © can be 
stated as follows. 

Theorem 1 (Analyticity): Let Assumptions |2]-|4] hold. Then, the following is true: 

i) /(•) is analytic on entire 8. 

ii) For each 6 E Q, there exist real numbers Sg G (0, 1), /ig € (1, 2], Mg £ [1, 00) such that 

\f{O')~fm<Me\\Vf{0')\r 

for all 6*' e 6 satisfying ||6i - 6i'|| < ^9. 
Theorem 2 (Convergence): Let Assumption [T] - 14] hold. Then, — \m\n^ooSn exists and satisfies Vf{6) = 
w.p.l on event A. 

Theorem 3 (Convergence Rate): Let Assumptions [T] - 2] hold. Then, 

l|V/(0„)|P = 0(7-^), \f{On) - fm = 0(7,7^), \\e„ 9\\ = 0(7-^) (4) 

w.p.l on A, where fi ~ fig and 

[1/(2 -A), ifA<2 

f — \ , p — fiinui{r,f}, g = min{(p — l)/2, r — 1}. (5) 

I 00, otherwise 

Proofs of the Theorems [T] - [3] are provided in Section |IV] 

In the literature on deterministic and stochastic optimization (notice that recursion ([1]) - ([3]) belongs to the class 
of stochastic gradient algorithms), the convergence of gradient search is usually characterized by gradient, objective 
and estimate convergence, i.e., by the convergence of sequences {V/(0„)}„>o, {f{0n)}n>o and {6'„}„>o (see e.g., 
O], im, on . II32I and references cited therein). Similarly, the convergence rate can be described by the rates at 
which sequences {V/(0,i)}n>o, {/(^n)}n>o and {9n}n>a converge to the sets of their accumulation points. In the 
case of algorithm ^ - (O, this kind of information is provided by Theorems |2] and [3] Basically, Theorem |2] claims 
that recursion ([T]l - (O is point-convergent w.p.l (i.e., the set of accumulation points of {6'„}„>o is almost surely 
a singleton), while Theorem [3] provides relatively tight bounds on convergence rate in the terms of Lojasiewicz 
exponent fig and the convergence rate of step-sizes {a„}„>o (expressed through r and {7n}„>o)- Theorem [T] 
on the other side, deals with the properties of the asymptotic log-likelihood /(•) and is a crucial prerequisite for 
Theorems |2] and [3] Apparently, the results of Theorems |2] and [3] are of local nature: They hold on the event where 
algorithm ([U - ([3]i is stable (i.e., where {On}n>o is contained in a compact subset of 6). Stating asymptotic results 
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in such a form is quite common for stochastic recursive algorithms (see e.g., ifTSl . ||25l and references cited therein). 
Moreover, a global version of Theorems |2] and [3] can be obtained easily by combining them with methods used to 
verify or ensure stabihty (e.g., with Q, ||9l or E5\ ). 

Various asymptotic properties of maximum likelihood estimation in hidden Markov models have been analyzed 
thoroughly in a number of papers in,|l5l,||a,||Ta,|[Il,|[l3-|l2a,|l2l,|l2l,||3^ (see also E Chapter 

12], lfT2l and references cited therein). Although these results offer a deep insight into the asymptotic behavior of 
this estimation method, they can hardly be applied to complex hidden Markov models. The reason comes out of 
the fact that all existing results on the point-convergence and convergence rate of stochastic gradient search (which 
includes recursive maximum likelihood estimation as a special case) require objective function to have an isolated 
maximum (or minimum) at which the Hessian is strictly negative definite. Since /(•), the objective function of 
recursion ([TJ - is rather complex even when the observation space is finite (i.e., y = {1, . . . , Ny}) and iV^;, 
Ny, the numbers of states and observations, are relatively small (three and above), it is hard (if possible at all) to 
show the existence of isolated maxima, let alone checking the definiteness of V'^/( ). Exploiting the analyticity 
of /(•) and Lojasiewicz inequality. Theorems |2] and |3] overcome these difficulties: They both neither require the 
existence of an isolated maximum, nor impose any restriction on the definiteness of the Hessian (notice that the 
Hessian cannot be strictly definite at a non-isolated maximum or minimum). In addition to this, the theorems cover 
a relatively broad class of hidden Markov models (see the next section). To the best of our knowledge, asymptotic 
results with similar features do not exist in the literature on hidden Markov models or stochastic optimization. 

The differentiability, analyticity and other analytic properties of the entropy rate of hidden Markov models, a 
functional similar to the asymptotic likelihood, have been studied thoroughly in several papers llT3l . lfT4l . ifTSl . 
Il29l . Il30l . Il35l . The results presented therein cover only models with discrete state and observation spaces and 
do not pay any attention to maximum likelihood estimation. Motivated by the problem of the point-convergence 
and convergence rate of recursive maximum likelihood estimators for hidden Markov models, we extend these 
results in Theorem [T| to models with continuous observations and their likelihood functions. The approach we use 
to demonstrate the analyticity of the asymptotic likelihood is based on the principle of analytical continuation and 
is similar to the methodology formulated in |13|. 

III. Examples 

In this section, we consider several practically relevant examples of the results presented in Section |ll] Analyzing 
these examples, we also provide a direction how the assumptions adopted in Section |ll] can be verified in practice. 

A. Finite Observation Space 

Hidden Markov models with finite state and observation spaces are studied in this subsection. For these models, 
we show that the conclusion of Theorems [T]- [3] hold whenever the parameterization of candidate models is analytic. 
Let Ny > 2 be an integer, while y ~ {!...., Ny}. Then, the following results hold. 
Proposition 1: Assumptions [3] and |4] are true if the following conditions are satisfied: 
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i) For each x,x' E X, y E y, rg{y\x' , x) is analytical in 6 on entire 0. 

ii) rg{y\x',x) > for all 6 E Q, x, x' E X, y e y. 

Corollary 1: Let Assumptions [T] |2] and the conditions of Proposition [T] hold. Then, the conclusions of Theorems 

□ -[3] are true. 

The proof is provided in Section [V] 

Remark: The conditions of Proposition [T] correspond to the way the candidate models are parameterized. They 
hold for the naturafl exponential^ and trigonometrij^ parameterizations. 

B. Compactly Supported Observations 

In this subsection, we consider hidden Markov models with a finite number of states and compactly supported 
observations. More specifically, we assume that 3^ is a compact set from R''". For such models, the following results 
can be shown. 

Proposition 2: Assumptions [3] and |4] are true if the following conditions are satisfied: 

i) For each x,x' E X, ro{y\x' ,x) is analytical in {9,y) on entire 0x3^. 

ii) re{y\x' ,x) > for all 9 E <d, x,x' E X, y Ey. 

Corollary 2: Let Assumptions [U |2] and the conditions of Proposition |2] hold. Then, the conclusions of Theorems 

□ -[3] are true. 

The proof is provided in Section [V] 

Remark: The conditions of Proposition|2]are fulfilled if the natural, exponential or trigonometric parameterization 
(see the previous subsection) is applied to the state transition probabilities {p0{x'\x)}x.x'eX7 and if the observation 
likelihoods {q0{-\xy} ^i^x are analytic jointly in and y. The later holds when {q0{-\xy\x(^x are compactly truncated 
mixtures of beta, exponential, gamma, logistic, normal, log-normal, Pareto, uniform, Weibull distributions, and when 
each of these mixtures is indexed by its weights and by the 'natural' parameters of its ingredient distributions. 

' The natural parameterization can be defined as follows: 9 = [ai,i ■ ■ ■ ck]Vx,]Vx/3i,i ■ ■ ■ Pn^ >^y 1"^ Pei^' 1^) = <^x,x' < Qeivl^) = Px,y 
for x,x' £ X, y e y, while is the set of vectors [q:i,i ■ ■■aM^,N^ /3i,i ■ ■■ l3N^,NyV € (0, l)^^(^^+^v) satisfying "i.i = 

"yifj^l fix,l = 1 for Ef'^h X a X. 

^ In the case of the exponential parameterization, we have 9 = ■ ■ ■ ajv^.iVx /3l,l ■ ■ ■ PN^,Ny]'^ , and 

for x,x' e X,y e y, while = R'^A^^+'^y). 

^ The trigonometric parameterization is defined as 9 = [oi,i ■ • • «jVa. ,jVx ■ ■ ■ /^jv^ .jVy ]^ and 

x'-l JVx 

pg{l\x) = cos^ ax,i, pg{x' \x) = cos^ a^.^, Y[ ^^^^ <^x,l, Pe{Nx\x) = Ylsia^ a^^i, 

1=1 1=1 
y-i Ny 
qe{l\x) = cos^ I3x,i, qe{y\x) = cos^ f^x,y Y\ ^^^^ l^x,l, = sin^ /3^^i 

1=1 1=1 

for X £ X, x' £ X \ {l,Nx}, y G ^ \ {1, Af^}, while = (0, 7r/2)^-(^-+^!'). 
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C. Mixture of Observation Likelihoods 

In this subsection, we consider the case when the observation likelihoods {qg{-\x)}x£X are mixtures of known 
probability density functions. More specifically, let da > 1, Np > 1 be integers, while A C M''° is an open set and 

S = I [/3i,i • • • PN^,N,f G (0, 1)^-^" : P-.k = 1 for each xeX^. 

We assume that the state transition probabilities and the observation likelihoods are parameterized by vectors a ^ A 
and [3 ^ B (respectively), i.e., = Pa{x'\x), q0{y\x) = qfj{y\x) fox a ^ A, (3 ^ B, 9 ~ [a"^/?^]^, x, x' £ X, 

y E y. We also assume 

Nf, 

Qi3{y\x) = 'Y(3x,kfk{y\x), 
fc=i 

where j3 = [Pis ■ ■ ■ I3n^,Np]'^ e B, x e X, y e y, while {fk{-\x)}x(zx,i<k<Nf, are known probability density 
functions. 

For the models specified in this subsection, the following results hold. 

Proposition 3: Assumptions |3] and |4] are true if the following conditions are satisfied: 

i) For each x,x' G X, pa{x'\x) is analytical in a on entire A. 

ii) Pa{x'\x) > for all a £ A, x,x' £ X. 

iii) V(y) > and / log2 i;{y')Q{dy'\x) < oo for all x G X, y e y, where ^(y) - E.ex Eff i fkiy\x). 
Corollary 3: Let Assumptions [U |2] and the conditions of Proposition [3] hold. Then, the conclusions of Theorems 

[T]-[3]are true. 

The proof is provided in Section |V] 

D. Gaussian Observations 

This subsection is devoted to hidden Markov models with a finite number of states and with Gaussian observations. 
More specifically, let da and A have the same meaning as in the previous section, while 3^ = R and 

B= {[Xi---Xn, pLi • --AiivJ^ e (0,oo)^- X R^- : ^ K for x ^ x' ,x,x' G X] . (6) 

Similarly as in the previous subsection, we assume that the state transition probabilities and the observation 
likelihoods are indexed by vectors a A and 13 £ B (respectively). We also assume 

9/3(y|a;) = ^/Khiexp{~\x{y - l^xf)-, 

where (3 =[Xi - ■ ■ /^i • ■ ■ IJ-nJ\^ e B, x e X, y e y. 

For the models described in this subsection, the following results can be shown. 
Proposition 4: Assumptions |3] and |4] are true if the following conditions are satisfied: 

i) For each x,x' e X, pa{x'\x) is analytical in a on entire A. 

ii) Pa{x'\x) > for all a £ A, x,x' £ X. 
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iii) / y'^Q{dy\x) < oo for all x £ X. 

Corollary 4: Let Assumptions [T] |2] and the conditions of Proposition |4] hold. Then, the conclusions of Theorems 
[T]-[3]are true. 

The proof is provided in Section [Vl 

Remark: Unfortunately, Proposition |4] and Corollary 2] cannot be extended to the case B = (0,oo)^=" x R^^, 
since the models specified in the subsection do not satisfy Assumption |4] without the condition A^, ^ X^' for x ^ x' 
(which appears in (l6]l)0 However, this condition is not so restrictive in practice as B is dense in (0,oo)^"= x 
and provides an arbitrarily close approximation to (0, oo)^^ x W^^. 

IV. Proof of Main Results 

A. Optimal Filter and Its Properties 

The stability properties (forgetting and ergodicity) of the optimal filter (|2]l, its derivatives (|3]l and its analytical 
continuation (to be defined in the next paragraph) are studied in this subsection. The analysis mainly follows the 
ideas and results of II2TI . ll22l and ||231 . The results presented in the subsection are an essential prerequisite for the 
analysis carried out in Subsections IIV-BI and IIV-CI 

Throughout this subsection, we rely on the following notation. denotes the set 

Q^^ ={ue [0,00)^- :e^u> 1/2}, 

where e = [1 • • • 1]^ G R^^ (Q^^ can be any compact set from [0,oo)^"' satisfying ^ Q^", intP^" C Q^", 
but the above one is selected for analytical convenience). For n > m > and a sequence y = {yn}n>o from 
3^, yrn:n dcnotcs finite subsequence (y„, . . . ,y„). For u e [0,oo)^- \ {0}, w G V G W^oy-N^^ n > jn > 

and sequences i9 = {i?„}„>o, V = {'7n}n>o. V = {yn}n>i from 8, C^", y (respectively), let G';5''™(u) = u, 
G^;^{w) = w, ff;';™(M,V") = V and 

G™;;+i(w) - G^„+,(G^-(w),y„+i), 
H^:;^\u,V) = H^„^,{G7:y{u),H^,y{u,V),yn+,) 

^Let ha,y,u{fi) = e'^Rg{y)u for a & A, (3 & B, 9 = y &y,u & -p^- . Obviously, for any a & A, y & y , u & T'^-, 

ha,y,u{-) has a unique (complex-valued) analytical continuation, which can be defined as 

ha,y,u{b) = ^ ^l^, /TTCXp{-l^,{y - m^,f)pa{x'\x)Ux 

where b = [h- ■ - In^ m\ ■ ■ ■ nif^^]'^ e C^-*^^ . Let /3 = [Ai ■ ■ ■ Ajy^ ■ ■ ■ e (0, 00)-'^^ X K-*^^ be any vector satisfying A^; = 

for some x 7^ x' , x, x' € X. Then, it is not hard to deduce that there exist a&A, y(zy,u(^ ■pN^ (depending on (3) such that ha.y,u{-) 
has a zero in any (complex) vicinity of /3. Since the zeros of the analytical continuation of Rg(y)u would be the poles of the analytical 
continuation of Gg{u,y), it is not possible to continue Gg{u,y) analytically in any vicinity of point {9,u), where 6 = Hence, 
Proposition |4] and Corollary |4] cannot be extended to the case B = (0, 00)^^ X R^=". 
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(Ge{u,y), G,j{w,y), He{u,V,y) are defined in Section If -d = {d}n>o (i.e., i^n ~ S), we also use notation 
G^-H = G"^--;^{u), H^^]y^{u,V) = H--;;^(u,V), as well as GO^«(w,yi:„) = G%l{u), Hf^\u,V,v^.,,,) = 
H^ly{u, V). Similarly, if = {?7}„>o (i.e., ??„ = 77), we rely on notation (^"^"(u;) = Gl^-^{w) and G'°'"(u;, yi:„) = 
G^'y(w). Then, it straightforward to verify 

G7;:i\w)^G'^--{G^:;:{;{w)), 

H^^:;\u, V) = i/^-(GS';^^u), (w, v)) 

for each u e [0,cx))^- \ {0}, w G C^-, F G M'''"'^-, < m < fc < n and any sequences -d = {z9„}„>o, 
^7 = {'?n}n>o. 2/ = {yn}n>i from 0, C^" , y (respectively). Moreover, it can be demonstrated easily 

G^^"(«;, 2/i:„) = G", '=(G^^"-'=(u;, yi.„_fc), y„-fc+i:„) (7) 

for all T] E C^", w € C^=", < fc < n and any sequence y = {j/„}„>i from 3^. It is also easy to show 

(Gr(",2/l:n),^e^"(«,^^,yi:n),yn+l) (V^G^ ) yi:n) ( V„</)e ) (G^ yi:n) , y„+l ) 

+ Ve(0e(Gr(",yi:n),yn+l)) (8) 

for each 9 e u e [0,oo)^=" \ {0}, V G ^d.g>iN^^ n> Q and any sequence y = {yn}n>i from y ((f>g{u,y), 
Fe{u,V,y) are Section [III (V„Gg'")(u, y), (VeGg'")(u, y) denote the Jacobians of Gg'"(it, y) with respect to u, 
6, while {Wu(f>e){u,y) stands for the gradient of (t)g{u,y) with respect to u). 

Besides the previously introduced notation, the following notation is also used in this section. For u e [0, oo)^^ \ 
{0}, n > m > and sequences i9 — {i9„}„>o, y = {yn}n>i from 0, y (respectively), let A''^-^y{u) = / e 
(/ denotes a unit matrix) and 

A:^:^{u) = (V„G^„^J(GS';;^(u),y™+i) • • • (V„G^J(G™;;-i(w),y„). 

Then, it is easy to demonstrate 

n-l 

H7:;\u, V)^V AJ-^{u) + ^(V,G,,,J(G;";;(«), y.+i) <^^"(G-+i(u)) (9) 

i—m 

for each u g [0, oo)^^ \ {0}, V G R'^»^^==, n > m > and any sequences -d = {i9„}„>o, y = {yn}n>i from 6, 
y (respectively). 

In this subsection, we also rely on the following notation. Sz and Sq denote sets Sz — X x y x V^" x ^^.e^^^ 
and ^ X X y X V^^ . For e 6, Pe{-, ■) and Pe{-, ■) are the transition kernels of Markov chains 

{x„+i, r„+i, Gg^"(?/, Yi:„), ri:„)}„>o and {x„, r„, Gr(«, Fi:„), V, yi:„)}„>o 

(respectively), while ne(-, •) and Iig{-,-) are the transition kernels of Markov chains 

{-^n+l, i^n+l, Gg "(M, Fi:„)}„>0 and {XmYn,Gl'^{u,Yi;n)}n>0 
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(notice that Pg{-,-), Pg{-,-), Ile{-,-), Ilg{-,-) do not depend on u, V). For 9 E Q, z = {x,y,u,V) E Sz, C = 
(x, y, u) £ Sq, let 

Fe{u,V,x)^E{Fe{u,V,Y2)\Xi^x), 
(j)e{u,x) = E((l)e[u,Y2)\Xi = x) 

while 

F{e,z)^Fg{u,V,v), F{0,z)^Fg{u,V,y), ^{6,0 ^ Mu,y), <^(0, C) = 0eK y)- 
Then, it is straightforward to verify 



=E (Fe(Gr(w,Yi:„),i70^"(M,V^,ri:„),X„)|Xi =2/) 



= (F"-iF) (^0, ( X, y, Gg{u, y), Hg{u, V, y) ) 
,0 =E{MG°e'"iu.Yi:n).Y^+i)\Xi=x,Y,=y) 
=E (^^e{G''g''{u,Yi.,^),Xn)\Xi = x,Yi = y 
= {IV'-'~^)(6,{x,y,Ge{u,y)) 



for all € 8, z = (x, y, u, F) G Sz, ( ~ {x, y, u) E Sq, n > 1. It can also be concluded 

iogp^Hn,...,r„+i) 



E 



n + 1 



——J2{Il^-^4,)(^9, ( X, y, Ge{ug,y) )) 



Xi ^ x,Yi = y 

Xi = x,Yi = y 

(l)e{ue,Yi) 
n+1 



(10) 



(11) 



(12) 



for each E S, C = {x, y, u) E Sq, n > 1, where ug [P{Xf = !)■■■ P{Xf = N^)f. 

Lemma 1: Suppose that Assumption |4] hold. Let Q C be an arbitrary compact set. Then, there exist real 
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numbers Si^q e (0, 1), Ci^q E [1,oo) such that 

\M^,x)\<C\.Q, (13) 

\\Fg{u,V,y)\\<C,^Q^Q{y)(l + \\V\\), (14) 

\\Feiu,V,x)\\<CuQ{l + \\V\\), (15) 

\^g>{u',x) - 4>e"{u",x)\ < C\Q{\\e' - 9"\\ + \\u'~ u"\\), (16) 

\^^,iw\y) - $.r,"{w",y)\ < Ci,QVQ(y)(h' - V"\\ + \W- w"\\), (17) 
\\Fe,{u',V',y)-Fe-,{u",V",y)\\ 

< C^My)il + \\V'\\ + \\V"me' 9"\\ + \\u' u"\\ + \\V'- V"\\), (18) 
\\Fg,iu',V\x)-Fe--iu'\V",x)\\ 

< Ci,q(1 + \\V'\\ + \\V"\\)i\\9' - 9"\\ + \\u'- u"\\ + \\V'- V"\\) (19) 



for all 9, 9', 9" e Q, v',v" e Vs,^q{Q), u,u',u" e w',w" e V,V',V" e R'^-'X^-, X e X, 

y € y (V'q(') is specified in Assumption |4]i. 

Proof: Let 6i,q = Sq/2 (Sq is defined in Assumption |4|l. Then, Cauchy inequality for analytic functions (see 
e.g., Il37l Proposition 2.1.3]) and Assumption |4] imply that there exists a real number Ci^g £ [l,oo) such that 

max{||V(^,„)(^^(w,y)||, |iV(^ ,„)0,,(u;,j/)||} < Ci^Q^pqiy) 

for all 7] E VsiqiQ), w £ V5jq(7^^"=), y G y (V(^ „,), denote the gradient and Hessian with respect to 

{ri,w)). Consequently, there exists another real number C2.Q G [l,oo) such that 

max{\\$.rj'{w',y) - (t),y>{w",y)\\,\\W^$rj'{w',y) - V„0,," (w", 

<C2,QMy)i\\v'-v"\\ + \\w'-w"\\) 

for any rj^rj" G Vs^^^iQ), w',w" G Vs,,,{V^''), yey. Therefore, 

\\Fe{u,V,y)\\ <\\VeMu.y)\\ + \\VuMu,y)\\\\V\\ 
<Ci,Q^Q{y){l + \\V\\), 

\\FB'{u'y,y)-Fe..{,u"y\y)\\ <|| Ve0eKw^ y) " Ve'/'e'^"", y)ll + li V„</>e' 2/) " V„</)e''(u'', y)|| || V^'ll 

+ \\Vu(Pe''{u\y)\\\\V' ~V"\\ 
<C'2,Q^Q(y)(l + \\V'\\ + \\V"m\9' 9"\\ + \\u' - ^"11) 
+ C,,Qi,Q{y)\\V' -V"\\ 



for each 9, 9', 6" e Q, u,u' ,u" £ V,V' ,V" e R'^"^^-. We also have 

\\Fe{u,V,x)\\<Ci,Q{l + \\V\\) J i;Q{y)Q{dy\x) 

\\Fg,iu',V',x)-Fe"iu",V",x)\\ 

< {C,.Q + (72,q)(1 + lly'll + \\V"m9' 9"\\ + \\u'- u"\\ + \\V'- V"\\) I ^PQ{y)Q{dy\x) 

for all 6*, 6*', 61" e Q, u,u',u" £ V,V',V" e R^oxn^^ x e X. Then, it can be deduced that there exists a 

real number C^q e [l,oo) such that (O - (Hill hold for each 9, 9', 9" G Q, r]',r]" e Vs^ ^iQ), u,u',u" £ 
w',w" £ Vs^q{V^^), V,V',V'' G K'^''^^^ x£X,y£y. ■ 
Lemma 2: Suppose that Assumption |4] hold. Let Q C O be an arbitrary compact set. Then, there exist real 
numbers (52, q £ (0, 1), C2,q £ [l,oo) such that 

||V,G,(w,2;)|| <C2,Q, (20) 

\\He{u,V,y)\\<C2,Q{l + \\V\\), (21) 
max{ \\G^,{w\y)~G^.[w",y)U\V^G,y{w\y)-V^ G^.. {w" ,y)\\} 

<C2,q{U ~v"\\ + \\w' -w"\\), (22) 

\\Hg,{u,V,y)-He->{u,V,y)\\<G2^Q{l^\\V\\)\\9'^9"\\ (23) 

for all 9, 9', 9" £ Q, r],v',v" e T^^.qCQ), w G w,w',w" £ Vs,_q{V^^), R^«^^-, y£y. 

Proof: Let 62,q = nim{6Q/2, Si,q}. Owing to Cauchy inequality for analytic functions and Assumption]?] 
there exists a real number Ci,q £ [l,oo) such that 

max{|lV(^^^)G^(u;,y)||,||V^^,^)G;;(u;,y)||} < (7i,q 

for any rj £ Vs^ q (Q)^ ^ G ^^2,q {P^")^ V ^ y (^^(^7 v) stands for the fc-th component of Gr^{w, y)). Consequently, 
there exists another real number G2.Q £ [l,oo) such that 

max{||G„,(u;',zj) - G^.. 2/)IMI V,A' K> 2/) " V^G^' K, y)|| , || V„G^- K, y) - V.^Gr,"iw" ,y)\\} 

<C2,q{\W-v"\\ + \W-^"\\) 

for all 77', 77" e Vs,_q{Q), w',w" £ (7'^="), y £ y. Therefore, 

\\Heiu,V,y)\\ <\\VeGe{u,y)\\ + \\VuGeiu,y)\\\\V\\ 
<Ci,QN,il + \\V\\), 

\\He' (u, V, y) - Hg„ {u, V, y)\\ <\\VeGe' {u, y) - VeGe- {u, y)\\ + || V„Ge' (u, y) - V„Ge" (u, y)\\ \\V\\ 

<C2^q{1 + \\V\\)\\9' -9"\\ 

for each 9, 9', 9" £ Q, u £ V'^^, V £ R'^«x^-. Then, it is clear that there exists a real number G2,q £ [1, 00) such 
that (EOli - (|23]i hold for all 9, 9', 9" £ Q, 77,77', 77" £ V5,_ ,^{Q), u £ w,w',w" £ Vs^ q{V^'^), V £ R^oxn^^ 
y£y. m 



15 



Lemma 3: Suppose that Assumptions [3] and |4] hold. Let Q C 6 be an arbitrary compact set. Then, the following 
is true: 

i) There exist real numbers Si^q E (0, 1), C3.Q £ [l,oo) such that 

(24) 



\\A7:^{u')-AJ;^{u")\\<C,,Qe-^' 



(25) 
(26) 



, y ; if,y 

for all u,u',u" e V^^, w',w" e Q^^, n > m > and any sequences 1? — {i?,i}ti>o. V = {yn}n>i from 
Q, y (respectively), 
ii) There exist real numbers £2.q G (0, 1), C4.Q £ [l,oo) such that 

\\H^,ly"{u,V)\\<C,^Q{l + \\V\\) (27) 
- < C,,QerQ^{\\u' - u"\\il + \\V'\\ + \\V"\\) + \\V' V"\\) (28) 



for all u, u\ u" e V^^, V, V , V" G 



> m > and any sequences i? = {^n}n>o, V = {yn}n>i 



from Q, y (respectively). 

Proof: Using |f 36l Theorem 3.1, Lemmas 6.6, 6.7] (with a few straightforward modifications), it can be deduced 
from Assumption [3] that there exist real numbers ei g e (0, 1), Cs^q G [1, oo) such that (|24] |. (IZSl l and 



hold for all tt , u' , u" G P^^, w', w" e [0, oo)^"' \ {0}, n > m > and any sequences i9 — {i?n}n>o, 2/ = {?/«}«>! 
from Q, J^lll Since 

^ \\w'~w"\\{e^w") + \\w"\\\e'r{w'-w")\ ^^^^^ ,^,„ , 
[e^ w')[e^ w") 

for any w',w" € Q^^, we have that ( l24b is satisfied for all w',w" e Q^^, n > m > and any sequences 
■»9 = {■dn}7i>o, y = {2/ri}n>i from Q, 3^. Hcncc, (i) is true. 

Now, we shaw that (ii) is true, too. Let i9 = {i?,i}„>o, y = {yn}n>i be arbitrary sequences from Q, y 
(respectively). As a consequence of Lemma |2] (i) and (|9]l, we get 



^ To deduce this, note that u, V, yo-.n, G^^'^{u), A^^'!y(u)V have the same meaning respectively as quantities fi, jl. j/", Fg{^,'y"), 
Gg{fj,, il,y") appearing in |36|. 

Inequality i26i can also be obtained from |20l Theorem 2.1] or (23, Theorem 4.1]. Similarly, (24), (25) can be deduced from fT9l 
Lemmas 3.4, 4.3, Proposition 5.2] (notice that G^''y{u), A'^'^{u) have the same meaning respectively as Mm.n, V[Mm,,n, Pm] specified in 
(19] Section 5]). 
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for all u e V^^, V G R'^e^^^, n > m > 0. Due to the same arguments, we have 

m:^{u',v')~H^^':^{u",v")\\ 

<\\A^l:;iu') - A"/:i\u")\\\\v'\\ + \\A^i^:^{u")\\\\v' - v"\\ 

+ E ll(V.G^,,J(GS';^(«'),2/.+i) - (V.G^.+J(G^;;(u'0,y.+i)||P:+i^«(G-+i(uO)|| 

i—'ni 
n-1 

i—7n 

n-1 

<G3.Qe",Q™r'|||hi' - u"\\ + Ga.ge^'.Q™!!^' - V"\\ + C^^qC.^q ^ <g "'HGS'.^^K) " GS^yK) 

i—m 



for each u' , u" G T'^", V , V" G R'^e^^-, n > m > 0. Then, it is clear that there exist real numbers e2,Q € (0, 1), 
G4,Q e [1, oo) such that dST), (|28]l hold for all w, u', u" e V , V" € R-ifX^- and any sequence -d = {i?}„>o, 

y = {yn}n>i from Q, y (respectively). ■ 
Lemma 4: Suppose that Assumptions [3] and |4] hold. Let Q C 8 be an arbitrary compact set. Then, there exists 
a real number Gs^g G [1, oo) such that 



l|Gg;"^(«)-G°;r^H||<G5.Q||0'-0"||, 

(u,F) - i?ry(u,F)|| < Gs.qII^?' - d"\\{\ + ||F||) 



(29) 
(30) 



for all 6*', 61" e Q, u e -p^% G 



> 1 and any sequence y ~ {?/«}«> i from 3^. 



Proof: Let Cq — C2^QC-i^QC\Q, while y — {j/„}„>o is an arbitrary sequence from y. It is straightforward 
to verify 



G°;:;(u) - Gg;r^(u) = ^ (G^^."^(Gg;j,^(u)) ~ G'^i^"(G°;j+i(u))) , 



(31) 



ri-l 



Hl-^(u, V) - H^il^Ju, V) = J2 [H^'^yiG^^yM, ^",«(", V)) ~ Hi+^''(G^^^in),H^;i^(u, V))) (32) 

4=0 

for all e\ 0" eQ,ue V^^, F G R''"''^-, n > 0. On the other side. Lemmas |2] and |3] yield 

\\GV,y{Gl,ly{u)) - G^^"(G^;^+i(u))|| = ||g^^" (G^^;'(Gg;j^^(u))) - G^^" (G^^^(G2;j^^H)) 



n — 2 — 1 



Q 



Gi;!^iG'p.y{n))~G-f^(Gp^^{n)) 



n—i—1 1 1 nf nif I 



(33) 



for any 6*', 9" e Q, u e V'^\ V G W^"^^'', < i < n. Using the same lemmas, we also get 

- HlVy' {G'ei^y iG'^,y(^^H-^^ {GpJu)),H",;fJu,V)))\ 
<G4,Qe^:Q-' \\G'ef^yHG°sly{u)) - G'^v^+J(G°;?,^(^))| 

■ (l + l^^'i' {GpJu)),H',;fJu,V))\\ + jH-^^ {GpJu)),H^;i^iu,V)) 

<3C2,qC|,q£'--1||^?' - e"\\{l + \\V\\) + C2,QC,,Qe-;Q^-'\\9' 9"U1 + WH^ifJu, V)\\) 
<5CQS-;Q^-'W-9"\\il + \\V\\) (34) 

for each 6', 9" e Q, u e V^'', F G R'*"''^-, < i < n. Combining (EB - (O, we conclude that there exists a 
real number Cg^g G [l,oo) such that (O, ^ hold for all 9', 9" e Q, u e "P^s V G R'^"^^-, n > 1 and any 
sequence y ^ {y„}„>i from y. ■ 
Lemma 5: Suppose that Assumptions |2] - 14] hold. Let Q C 8 be an arbitrary compact set. Then, the following 
is true: 

i) /(•) is well-defined and differentiable on Q. 

ii) There exist real numbers ea.g G (0, 1), Ce.g G [l,oo) such that 

\\{P-F){9,z) Vf{9)\\ < Ce^QslQil + \\Vr), 

\{n-^)i9,C)~fm<Ce,QelQ 

for all 6* G Q, 2; = (x, y, u, V) G Sz, C — {x, y, u) G 5^, n> 1. 
Proof: Using 1361 Theorems 4.1, 4.2] (with a few straightforward modifications), it can be deduced from 
Lemma [T] that there exist functions g : 8 —> R'''', ?/; : 8 — > R and real numbers ea.g G (0, 1), Cg^g G [l,oo) such 



that 



\\{P-F){9,z)~g{9)\\<Ce.QslQ{l + \\V\n 
\iIl-4')i9,0~m\<Ce.QelQ 



(35) 
(36) 

for all G Q, z = (a;, y, u, V) G Sz, C — {x, y, u) G 5^, n > iQ Since E\(j}0{ug,Yi)\ < oo for any 9 e Q (due to 
Assumption it follows from ( fTSI i. (l36T l that /(•) is well-defined and identical to on Q. On the other side. 
Lemmas [T] |3] yield 

\\Fg{Gt-^{u,yi..^),H"s-^{u,V,yi..r.),yn+i)\\ <Ci,QVQ(y„+i)(l + IjiJ^K 2/i:n)ll) 

<2Ci,QC4,QVQ(yn+l)(l + 11^11) 



'The same result can also be obtained from [20' Theorem 5.4] 
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for each 9 £ Q, u £ V^"", V £ and any sequence y = {yn}n>i from y. Then, Assumption S] gives 

E ( |lFe(Gg^"(u, Yi.,„),H^--'\u, V, yi:„), r„+i)|| \Xi=x,Yi= y) 

< 2Ci,qC4,q(1 + lll^ll) max / ijQ{y')Q{dy'\x') < oo 

for all 6 E Q, u E V^", V E M''<'^^% x E X, y E y. Consequently, the dominated convergence theorem and ([8]), 
(Unil, (HB imply 

Ve(n"-V)(^, C) {We (MOf'^iu, Y,.,n), Yn+i)) \X,=x,Y,^ y) 

(Fe(G°^"(u, Yi:„), ijO'X", 0, yi:„), r„+i)| =x,Yi= y) 
-(P"-iF)(0,(C,O)) (37) 

for any 6 E Q, C, ^ {x, y, u) e n > 1 (here, stands for de x Nx zero matrix). As (n"(/))(6', () and {P'^F){6, z) 
converge (respectively) to ■i/j{9) and g{9) uniformly in 6* e Q for each z E Sz, C € (due to ( flOb . ( fTTT i. ( |35] ). 
(O), it follows from ( l37b that Part (i) is true. Part (ii) is then a direct consequence of ( fTOl i. ( fTTT ). ( |35] ). ( l36b . ■ 
Lemma 6: Suppose that Assumptions |2] - 14] hold. Let Q C 8 be an arbitrary compact set. Then, there exists a 
real number Ct^q E [1, oo) such that 

\\{P"F){9',z) - (P"F)(0",z)|| < Cr,Q\\9' - 9"\\{1 + \\Vf) (38) 

for all 9', 9" E Q, z ^ {x, y, u, V) ESz,n> 1. 

Proof: Let Owing to Lemmas [T] [3] and |4] we have 

||P9'(Gg;"(u, 2/i:„), Hf,^{u, V, yi:„), yn+l) - 2/l:n), ^, yi:n), ^n+l)!! 

<Cl,QVQ(yn+l)(l + + \\Hf,? {u,V,y,..n)\\) 

■ {\\9' - 9"\\ + ||Gr(«,yi:n) - G°;r(u,2/i.„)|| + yi:„) - yi:„)ll) 

< 9Gi,QG4.QG5,QVQ(2/n+l)(l + II^ID'll^?' - ^"11 
for all 9' , 9" E Q, u E P^", V E R''*^^^, n > 1 and any sequence y ^ {?/„}„> i from y. Consequently, 

\\{W^F){9',z) - (^"F)(^^",z)|| <i?(||p,,(GrK2/i:n),i?r(«,^,yi:n),y„+i) 

- Pe"(Gg;r(u,2/i:„),i7,";r(w,t/,yi:„),yn+i)ll l^i ^ ^,Y, ^ y) 
<9Gl,QG4,QG5,Q(l + ||F||)2||^?'-^^"||max / VQ(y')Q(rfyV) 

x'£X J 

for each 6'',^" €(5,2:= {x,y^u,V) E Sz- Then, it can be deduced from Assumption |4] that there exists a real 
number Ct^q E [1, oo) such that ([38]l holds for all 9' , 6" E Q, z = {x, y, u, V) E Sz- ■ 
Lemma 7: Suppose that Assumptions |3] and |4] hold. Let Q C 8 be an arbitrary compact set. Then, there exist 
real numbers (^3 Q,e4,Q E (0, 1), G8,q E [1,oo) such that the following is true: 

i) G°;y(w) is analytical in {r\.,vj) on ^ (Q) x ^53^(7^^^) for each n > and any sequence y = {?/n}n>i 
from y. 
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ii) Inequalities 

- G?;SK)I1 < ^.g^glk' - W'w 

hold for all 77 e V53 ^ (Q), w, w' , w" G Vs^ ^ [V^^) and any sequence y — {y„}„>i from y {5q is specified 
in Assumption |4ll. 

Proof: Let y = {yn}n>i be an arbitrary sequence from 3^. Moreover, let kg — min{n > 1 : Cs^qe^q < 
while Si^Q = mm{6Q,5i^Q,S2^Q}, S2,q = 4.^'''^ C~q^ Si^q. 
First, we prove by induction (in k) that 

for all 77 e Vg^^{Q), w € 71 > 0, < fc < A:q. Obviously, ^ is true when fc = 0, n > 0, 

^ ^ Q ('S)' ^ ^ ^fc Q ('^^")- Suppose now that ^ holds for each ?7 e Vj, ^ (Q), G q ('^^"), n > and 
some < fc < fcg. Then, Lemma |2] implies 

<C2,Q(h-^^|| + ||G-+^■H-«ll) 

for any 6* e Q, 77 e t/^^ ^ (Q), u G P^-, w S T^^^ ^ {V^"-), n > 0. Therefore, 

d(G:;;;;+'=+i(«;),P^-) <G2,q (d{^M)+d{G^^Z^\w\V''-)) 

<2'^+'G',+'~52,Q 

<(2'=+2G^;+i - 1)52,Q < Kq 

for any € V^^, ^ (Q), w e V-^^ ^ (V^^), n > 0. Hence, ^ is satisfied for all V V's^ q (Q)' ^ ^ q (^^")' 
71 > 0, < fc < fcg. 

Let 4,Q = ^2.q/2. Since G^^-^^iw) = w and G;;;^+'=+1(w) = a,(G;j;^+'=(u;), y„+fc+i), it can be deduced from 
Assumption m and (|39]l that GJ^;y+''(if) is analytic in (77, w) on Vj^ ^ (Q) x Vg^ ^ {V^^) for each 7t, > 0, < fc < fcg 
(notice that a composition of two analytic functions is analytic, too). Due to Assumption |4] and ( [39] l, we also have 

\\G;]^+'+'H\\ = \\G,{G^-;+Hw),yn+k+i)\\ < Kq (40) 

for all 77 e V^^ (Q), w € (P^^), ri > 0, < fc < fcg (ivTg is defined in Assumption gll. As a consequence of 
Cauchy inequality for analytic functions and (|40] |. there exists a real number Gi g G [l,oo) depending exclusively 

on Kq, dg, Nx (Gi^Q can be selected as Gi^g = 4:{de + Nx)Kq/62 q) such that 

max{||V(,,^)G-+^7.)||, llV^^.^jG^+n^^)!!} < Gi,Q 

for any 77 e Vg,^^{Q), w E V"^^ ^("P^-), n > 0, < fc < fcg, 1 < / < iV^r iGTJi'y''{w) denote the l-th component 
of G^'y+'^(u))). Consequently, there exists another real number G2,q G [loo) depending exclusively on Kq, dg. 
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Nx such that 

max{||G:;7;+'^K) - G;;-+'{w'')\\,\\W^G;;l+'iw') - ^.^G;;-+'{w")\\} 

<C\Q{h' ~rj"\\ + \\w' ~w"\\) (41) 

for each 77',??" e Vg^ ^iQ), w',w" G n-3,«(^'^^)' > 0, < fc < fcg. 

Let Q = niin{(53.Q, 4^^C'^g£i q}. Owing to Lemma |3] (Part (i)), we have 

\\g::;+'''{u') - g;i;^'-{u")\\ < G,,Qe',%\\u' - u"\\ < ie,,Qmu' - u"\\ 

for all e e Q, u',u" £ [0,c»)^- \ {0}, n > 0. Therefore, || VuGg'^^'"^ < £i.q/2 for each 9 e Q, u e 
[0,00)^=" \ {0}, n > 0, which, together with dHJ yields 

W'^^G'^T'^'HW <iiv„G';;;+'^'=(u)ii + wv^G^^T'^^iw) - v„g;;;+'=«(u)|| 

<ei,Q/2 + G2,Q(||(?-77|| + 
for any 6 e Q, rj e ^ (Q), u G V^'^, w G Vj^ ^ ("P^-), n > 0. Consequently, 

||V^G:;:^+'«H|| <£i,Q/2 + G2,Q(d(77,Q)+d(«;,P^^)) <ei,Q 
for each r/ e ^ (Q), w e ^ ("P^-), n > 0. Thus, 

wd'^T^'^iw') ~ G:;;r'''(^")ii < iiv„G;^^''«(to' + (i - - w"\\dt < s^,q\\w' - w"\\ (42) 

Jo 

for all e V^^, JQ), e JP^^), " > 0. 

Let (55_Q = (1 — £i,g)'54,QG2"g. Now, we prove by induction (in i) that 

d(G°;tf«H,7'^^) <^4,Q (43) 

for each V ^Vgr^Q (<?)' ^ <, C''^" )' « > 0- Obviously, (gS]) is true when i = 0, ry G Vg^^ ^ (Q), w £ Vg^ ^ ("P^- ). 
Suppose that (|43]l holds for all V ^ V~s,^ ^ (<3). ^ ^ ("P^-) and some i > 0. Then, gB, dUll imply 

llG^:y^'"«M - g;5^'''+'^'«hii <iiG;'«^<^+'^''^(G°;;''''(«;)) - g;':«^'^+'^'«hii 

+ iig;';«^^^+'^'«h - g;';«^^'+'^'=«(w)|| 

<ei,Ql|G"t'^H-«||+G2,Q||^?-r;|| 

for any 6* e g, 77 e V^^^ ^ (Q), u e 7^^-, w £ ^ (T'^-). Therefore, 

d(G"J+'''^^(«;),7'^i < ei,Qd(G°:;''«H,7'^^) + G2,Qd(?7,Q) < £i,q5\q + Ga.g^s.Q = ^4.q 

for each rj e ^ (g), w eV-g^^ (V^^ ). Hence, dH holds for all rj e T/^^ ^ (g), «^ e V"^^ ^ « > 0. 

Let Ss^Q = mm{h^Q,k,Q}- As G^^%{w) - t« and C'^f'^^'H = G;':«^<*+')'« (G^J;;'" (u;)), it can be 
deduced from (|43] l that G^'^y (w) is analytical in (r],^) on V^^ q(Q) ^ ^Siqi'^^'') ^^^^ i > (notice that 
G'^^,§'-^"^^'^'"^ (w) is analytic in (77, w) on ^ (g)xF5^ ^ (T'^-) for any ^ > 0). Since G°^%{w) = &^T\G°,^'v'^ {w)) 
for i — [n/fcgj, we conclude from ( l43T l that G°'^(u>) is analytical in {ri,w) on V'^^qCQ) x ^54q(''^^'°) — 
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^53,q(Q) X ^53, q(^^") for all 71 > (notice that Gf,§'-"'"'^'^^ (w) is analytical in {r],w) on Vg^^{Q) x Vg^^(r^-) 
for any « > 0, < j < /cq). On the other side, ( [39] ), ( |43] ) yield 

d(G?-^H,7'^-) = d(G;':r"(G°:y«(^«)),7'^-) < 5~i^q = min{<5Q,<5i,Q,<52,Q} (44) 

for all 77 e 2 ^53,«(Q), € J^^^) 2 ^i3.Q (^'^==), 7i > and 7 = [n/fegj. 

Let e4,Q = ej/g"' ^s.Q = ^^Q^i}}- Owing to gUi, (US, we have 

iiG°:l;+'^'«(u;') - g":1;+'^'''K)ii -iiG;':r''+'^'''(G°:y'^K)) - g;':«^^^+'^'«(g°:;'«k'))ii 

<£i.Ql|G°r^(u;')-G°r^K)|| 
for any V & Vg, q (Q)' ^ "^^4 « C^^")' « > 0. Therefore, 

iiG°;;f«K)--G$;;;f«K')ii<£i,Qii^'-^«"ii 

for each t] E Vg^^iQ), w',w" G Vg^ ^{V^'-'), i > 0. Consequently, (gB, (gS]) yield 

<C2,QelQ\\w'-w"\\ 
<Cs,QelQ\\w' ~w"\\ 

for each rj e ^^^^(g) 2 Fi3,^(Q), w',w" e JT'^^) D V^53,q (^'^^). 77 > 0, i = [n/kgl (notice that 
^2,Q£i.Q = C2,Qe^ Q ^'^'^^Si^Q < G8,q£Jq). Then, it is cleai" that (53, g, e4,Q, G8,q meet the requirements of the 
lemma. ■ 



B. Analyticity 

In this subsection, using the results of the Subsection IIV-AI (Lemma |7]), the analyticity of the objective function 
/(•) is shown and Theorem[T]is proved. The proof is based on the analytic continuation techniques and the methods 
developed in lfT3l . 

Proof of Theorem [7} Let 



for 77 £ 



w G 



X E X, n>l. Then, using (|7]l, it is straightforward to verify 



v;^+i(77;,.x) =i?(i?((^,(G°;"(G,(77;,ri),r2:„+l),r„+2) 



Xl , X2 , 1'l 



(45) 
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for each 77 e C*", w e C^"', x e A", n > 0. It is also easy to show 

((^^(G", "K, Yi.„), r„+i) - <^^(G:;^"(eo, ri,„), r„+i) 



E 

ri-1 



(0^(G°^"K, ri:„), r„+i) - 0^(G°;"(eo, 



= x" 



k=l xeX 



Xk = X 



{p'^-\x\x")-T,{x)) 



S(0^(G^(eo,y„),y„+i)|X„ = x)(p"-i(x|x') -7r(a:)) 
Y S(0^(G^(eo,r„),r„+i)|X„ = -7r(x)) 



(46) 



xex 



for all ?7 e C'*^, w',w" G C^-, e X, n>l, where eo = [1 • • • l]^/iV^ G and p'=-i(x'|j:) = P(Xfe = 

x'\Xi ~ x), tt{x) — linifc^oo -P(-''^fe = x). On the other side. Assumption |2] implies that 7r(-) is well-defined and 
that there exist real numbers £6(0,1), Gg[1,oo) such that 



|p"(a;'|x)-^(x')| <Ge" 



(47) 



for each x,x' G X, n> 0. 

Let Q C 8 be an arbitrary compact set, while Si,q = min{(5Q, Ji g, (52,q, (53,q}, S2,q = (5i,q/2. Owing to 
Assumption m and Lemma |7] 0^(G°'"(u', ?;i:„), y„+i) is analytic in (77, w) on Vg^^^iQ) x ^SiqO^^'°) ^'^^^ 
n>0 and any sequence y ~ {yn}n>i from y. Due to Assumption |4] and Lemma |7] we also have 

\^rj{G'^-"'{w,yi:n),yn+l)\ < IpQiVn+l) 

for all 77 e Vg_^ q(*5)^ G q(''^^^)' ^ ™d any sequence y — {yn}n>i from 3^. Consequently, Cauchy 
inequality for analytic functions implies that there exists a real number Gi g G [l,oo) such that 

\\'^vMG°r,'"{w,yi..n),yn+l)\\ < Gi.qtAq (?/„+i ) 



(48) 



for each r/ G Vg^ q(*9)' ''^ ^ q^'^^'°'^' n>0 and any sequence y = {j/„}„>i from J^. Since 

E{i;QiYn+i)\Xi =x)< max / ^Qiy')Q{dy'\x') < 00 



(49) 



for all X G A", n > 0, it follows from the dominated convergence theorem and ( |48] | that ?/;^(u',x) is differentiable 
(and thus, analytic) in 77 on Vj^ ^ (Q) for any 71; G Vg^ ^ [V^''), n>Q. 
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Let eq — max{e4_Q, e}. Due to Lemmas [T] and |2l we have 

|0,(G°^"(«;^yl:„),y„+l)-<^,(G^^"(^«^yl:„),y„+l)| <Ci^^ (50) 
|(^„(G°;"-'-+i(«;,yfc:„),y„+i) - 0„(G?/"-"(w,yfc+i:„),2/„+i)| 

< Gi,QVQ(yn+i)l|G°;"-'=(G,(«;,yfc),y,+i:„) - G'^^^-^w.y k+i:n)\\ 

< Gi,QG8,Qe4_Q''i/'Q(2/„+i)||G^(w,?/fe) - w\\ (51) 

for each 77 G V^^ ^ (Q), w, w' , w" G V^^ ^ {V^^), n>l, 0<k<n and any sequence y = {yn}n>i from y. Using 
( l47T i. (|49] l - ( fSTT l. we deduce that there exists a real number C2,q G [1, 00) such that the absolute value of the each 
term on right-hand side of ( l46b is bounded by C2,qSq for any 77 £ Vg^ q ('S)' ^'j"^" ^ q(^^°^)' ^ 
n > 1. Therefore, 

|^;(«;',x')-^;;K,x")| <2G2,Qe^(n+l) (52) 
for all T] e ^ (Q), w' , w" G ^ (P^^), x', x" G A", n > 1. Consequently, (EH) yields 

- i^:;iw,x)\ < E(^\^^{G,iw,Y,),X2) - i:^^{w,x)\\x^ = x) < 2C2.Qe^(n+ 1) (53) 

for each 77 G V^^ (Q), w ^V^^^ ("P^-), x eX,n>l. Owing to there exists a function i/i : C*" ^ C 

such that x) converges to uniformly in (ry, w, a;) G V^^ ^ {Q) x ^ [V^'' ) x A". As the uniform limit 

of analytic functions is also an analytic function (see ||37l Theorem 2.4.1]), is analytic on Vg^ giQ)- 
other side, since 

$'S{u,x) ^ E {MG°e-''{u,Yi,n),Yn+i)\ = x) = E {{n''-^<j>){e, {x,Yi,u))\ X^ ^ x) 

for all 6* G 6, u G V'^'', x e X, n > I, Lemma |5] implies f{9) = tij{6) for any e Q. Then, it is clear that Part 
(i) is true, while Part (ii) follows from the Lojasiewicz inequality (see e.g., ifTTl . Il26ll . Il27ll ) and the analyticity of 

/(•)■ ■ 
As a direct consequence of [17, Theorem LI, Page 775] and Theorem [T] we have the following corollary: 
Corollary 5: Let Assumptions |2] - H] hold. Then, for any compact set Q C 8 and real number a G f{Q), there 

exist real numbers Sg^a G (0, 1), /iq^q G (1, 2], Mg^a € [1, 00) such that 

\f{0)-a\<MQj\Vfie)\r-- 

for all 6* G Q satisfying \ f{9) - a| < Sg^a- 

Remark: Obviously, if Q C {9' e M.'^" : \\9' - 9\\ < Se} and a = f{9) for some 9 G then /ig^^ and Mq^a 
can be selected as /ig^a — and Mg ^ — Alg (Sg, fiQ, Mg are specified in the statement of Theorem [T]). 

C. Decomposition of Algorithm (Q]) - 

Relying on the results of Subsection II V- Al (Lemmas [Tl - 16]|. equivalent representations of recursion ([B - © and 
their asymptotic properties are analyzed in this subsection. The analysis is based on the techniques developed in ^ 
Part 11]. The results of this subsection are a crucial prerequisite for the analysis carried out in the next subsection. 
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In this subsection, the following notation is used. For n > 0, let Zn+i = {Xn+i, i^i+i, Un, Vn), while 

— F{6„, Zn+i) — V/(0„), 

= a„(V/(^„))^6M 

€ = I (v/(^?„ + mn+i - On)) - vf{e.n)f{en+i - e^)dt 

and (j)n — 4'n + 4'n (^('i ■) is defined in the beginning of the previous subsection). Then, algorithm ([B - © admits 
the following representations: 

On+l =0n + CtnF{9n, ^n+l) 

(v/(^?„) + 60, ">o. 

Moreover, we have 

fiOn+l) - f{On) + an\\Vf{0n)f + </>„ 

for n > 0. We also conclude 

P{Zn+l G B\9o, Zq, . . . ,9n, Zn) — Pg^ (Z„, B) 

w.p.l for n > and any Borel-measurable set B C Sz (Pe{-, ) is also introduced in the beginning of the previous 
subsection). 

Lemma 8: Suppose that Assumptions |2]-|4] hold. Then, there exists a Borel-measurable function $ : 8 x 5^ ^ 
M'^" with the following properties: 

i) $(0, •) is integrable with respect to Pe{z, •) and 

F{e, z) - wf{e) = <f{e, z) ~ {p<s>){9, z) (54) 

for all 6* e e, z e S^- 

ii) For any compact set Q C 8 and a real number s G (0, 1), there exists a Borel-measurable function ^pq^s ■ 
Sz [l,oo) such that 

ma^{\\F{0,z)\\,\m0,z)\\,\\{P<S>){9,z)\\}<ipQjz), (55) 
||(P<i>)(0',2)-(F$)r,z)|| <^Q^.(z)||0'-0"||^ (56) 

Snp E(lfQ s{Zn)I{TQ>n} \ Zq^ z) <oo (57) 
ri>0 

for all 9, 9', 9" e Q, z e Sz, where 

TQ = inf{n > : 9„ ^ Q}. 

Proof: Let Q C 8 be an arbitrary compact set. Owing to Lemmas [T] and |5] there exists a real number 

Ci.Q € [IjOo) such that 

oo 

J2\\iP'F){9,z)~Vfm<Ci,QMy)i^ + \\Vr) (58) 

fe=0 
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for all 6* e 0, 2 = {x, y, u, v) G ({P°F){e, z) stands for F{9, z)). Consequently, E^o((^''^)(^' ^) ~ '^/(^)) 
if well-defined and finite for each 6 ^ Q, z £ Sz- We also have 

oo oo 

Y,iiP'F){9', z) - Vf{e')) - ^((P'^F)(0", z) - V/r)) 

n 

< E II {P'^FW^ ^) - {P'FKe", z)\\+n II v/(e') - V/(0")|| 

oc oo 

+ E ||(^'^^)(^',^)-v/(^?')||+ E ||(i^'^)(^",^)-v/(n|| 

A;— n+l k—n-\-l 

for each ^" G z G Sz, n> 1. Then, using Lemmas [5] and [6l it can be deduced that there exist real numbers 

eg G (0, 1), C2,Q e [1, oo) such that 

oo oo 
k=l k=l 

for all 6*', 9" £ Q, z = {x, y, u, V) £Sz,n>0 ({P^F){9, z) is defined as F{9, z)). 

Let Cq = max{C'i,Q, C'2,q}. Moreover, let NQ^s{t) = \s\ogt/ logigl for s,i e (0,1) and NQ^s{t) = for 
s £ (0, 1), t £ {0} U [1, oo). Then, it can be concluded that there exists a real number Kq^s G [1, oo) such that 

NQAt)+^Q'''^'^'^ <KQ,sf (60) 

for all t £ [0, oo). 

For 9 £ e, z = {x, y, u, V) £ Sz, let 

OO 

^9,z) = Y,{{P'^F){9,z)-\/f{9)), 

iPQ,s{z) = CQKQ,sMy)i^ + \\vf)- 

Since 

{PipQ,s){9,z)^CQKQAl + \\H9{u,V,y)f)E{i;Q{Y2)\Xi=x) <cx, 

for all 6* € 8, z = (a;, y, u, V) £ Sz, we deduce from (ISST i that $(•, •) is well-defined, integrable and satisfies ( [54] l. 
(I55]l (notice that (P$)(e', z) = E^i ^) - V/(6l))). On the other hand, dSDl, dSil imply 

||(P$)(0',z) - (F$)r ,z)|| < CqKqJI + \\Vr)\\9' - 9"r 

for any 9', 9" £ Q, z = {x, y, u, V) £ Sz (set n ^ NqA\\9' - 6'"||) in dSU). Thus, ^ is ti'ue for each 9', 9" £ Q, 
z = {x,y,u,V) £ Sz- 

Let 9 = {9n}n>a and Y = {Yn}n>i- Due to Lemma|3] we have 

<ACQKQ,,ClQi;Q{Y„+,)il + \\Vof) (61) 



<C2,Q{l + \\Vf)ie"Q+n\\9'-9"\\) (59) 
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for each n > (notice that Hqy{Uo, Vq) depends only on the first n elements of 6, and that 9i, . . . ,9n G Q is 
sufficient for dHTT l to hold). Consequently, 

a: J 

for all z = {x, y, u, V) G Sz, n> 0. Hence, ( fSTl ) is true for all z g Sz- ■ 
Lemma 9: Suppose that Assumption [T] holds. Then, there exists a real number s e (0, 1) such that 

Proof: Let p = (2 + 2r)/(2 + r), q (2 + 2r)/r, s = {2 + r)/{2 + 2r). Then, using the Holder inequality, 
we get 




Since 7„+i/7„ = 1 + q;„/7„ 0(1) forn -> oo and 

a« _ 7»+i - 7« < 7«+i \ ^ /■'^"+' ^ - J_ 7n+i A ^ 

^1 ~ 7?i " ;S V 7n / A. " 71 '">f V 7n / ' 

it is obvious that X^J^Lo Q^n^*7ri converges. ■ 
Lemma 10: Suppose that Assumptions [T|- 2] hold. Then, there exists an event A^'o such that P{Nq) — and such 

that Y.n=n an7,'iCn, Y.n=o "n^« ^"'^ Yln=o <^n converge on A \ A'^o- 

Proof: Let Q C O be an arbitrary compact set, while t is an arbitrary number from [0,r]. Moreover, let 

: O W^o ^ ''■'> be an arbitrary locally Lipschitz continuous function. Obviously, in order to prove the lemma, it 
is sufficient to demonstrate that X^^^o Y^n^=G^n converge w.p.l on (Xa^ai^^ G Q} (to show 
the convergence of ^nln^n, set f = r and = / for all 9 E 0, where / stands for de x dg unit 

matrix; to demonstrate the convergence of X^^o'^n' set f = and '^{9) = e{Wf{9))'^ for each 9 E Q, where 
e = [l---lf eR'^"). 

Let s E (0, 1) be a real number such that '^n^^^n < while 

r - n..nJii"-T-rpiii md')-'^iO")\\ ||v/(gO-v/(nH . n ^ ^\ 

Moreover, for n > 1, let 

i^l.n = ^i9n)iM9n, ^„+l) - {P^)iOn, 

V'2,« = ^'(0„)((P$)(0„, Z„) - (P$)((?„_i, Z„)) + (^'(^„) - *(0„_i))(P$)(0„_i, Z„), 
^3,n = *(en)(P*)(e„,^«+l). 

Then, it is straightforward to verify 

n n n n— 1 

^ a,,7*^'(6'j)^,, = ^ a,;7*^i,j: + X! "*7*^2,» + ^(a,+i7*+i " ai7*)'03,i - an7nV'3,n + ao7oV'3,o (62) 
for n > 1. 
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Owing to Assumption [T] we have 

an = + an(a,T+i ^ "rT^)) = 0(a„+i), 

7^1 - 7r« = 7,* ((1 + an/lnf - l) = o(a„7*,) 

as n ^ oo. Consequently, 

CXD OO 



n=0 Ji=0 

OO 



X! I"" 7^ - a«+i7^+il < X! ~ + X! I"" ~ "n+i|7^+i < OO. (64) 

n—O n—Q n—Q 

On the other side, as a consequence of Lemma [8] we get 

■ n\ I{TQ>n }) < '^CqEo^z {v'QA^n+l)I{TQ>n}) + ^CqEb^z {tpQ^^{Zn)I{rQ>n-l}) , 
{\^2.n\I{r^>n}) < ^CgEe,, ((^Q,, (Z„ ) 1 1 0„ - 0„_1 1 1 ^/{.^ >„} ) < 2C'Q<_i£;e,. (<^Q,s (^n)/{rQ >n- 1} ) 

for all 6' e 8, z G Sz, n > 1. Due to the same lemma, we have 

Ee,z (IV'3,nP^{TQ>n}) < CgEg^z ifQ.si^n+l)I{TQ>n}) , 

Ee,z {Wn\I{TQ>n}) < CqEe^z {\\dn+l - On\\'^I{TQ>n}) < Cga^Eg^z (^"+1 )^{tq >n}) 

for all 6* e e, z e 5^, n > 1. Then, Lemma [8] and (|63]) yield 



OO 



n>0 



( X!""'^"l^2,n|-f{rQ>n} j < SCq ( ^<_ia„7^ j SUpEe,^ {VQ,siZn+l)I{rQ>n}) < 
\n=l / \n=l / 

for any £ Q, z E Sz- On the other side. Lemma [8] and (l64b imply 

|a"7n - an+l7n+ll l'/'3,n|/{rQ>n} 



Vn=l 

OO 



- 1^ ~ "»+i7n+il sup ((^1 (Z„+i)/| >„}))^^^ < oo, 

\„=i / 

Ee,z '^al+lln+l\'4'3,nfl{TQ>n} ] < Cq i ^ a^+i7^*+l SUpi^e,^ (</^Q,s(^n+l)^{rQ>n}) < 
Vn=l / Vn=l / ">0 

Ee,z [ XI l'?^"l-^{rQ>n} I < I X I SUpi;e,^ (</'Q,.(^«+l)i^{rQ>«}) < OO 



Vn=0 / \Ji=0 

for each 9 £ Q, z £ Sz- Since 



Ee,z (^l,n/{rQ>„}l^n) = ^C^n) (£^8.. (*(0„ , 2'„+i ) |.F„) - (P*) (0„ , Z„) ) /{,^ >„} = 
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w.p.l for every 6* G 8, 2 G Sz, n > 1, it is clear that series 

C30 CXj CXD 00 

n—1 n—1 n—1 n—1 

converge w.p.l on H^oi^" ^ '3}' '^^'^ lii^^n-+oo ctnT^V's.n = w.p.l on the same event. Owing to this and 
(|62li, we have that I]^o '^»7n*(6'n)C« is convergent w.p.l on CC^oi^n e Q}. ■ 
Lemma 11: Suppose that Assumption [T] - 2] hold. Then, on A \ Nq, lim„^oo V/(6'„) — and lim„^oo /(^'n) 
exists. 

Proof: Let Q C be an arbitrary compact set, while lo is an arbitrary sample from HJ^oi^" ^ '3} \ ^0 
(notice that all formulas which appear in the proof correspond to this oj). Obviously, in order to prove the lemma, 
it is sufficient to show that lim„^oo /(^^n) exists and that lim„_^oo V/(6'„) = 0. 
Since Y^^=o 4'n converges and 

n—1 n—1 

^ a,\\vf{e,)f = f{e„) - f{e,) - ^ 

i=0 1=0 

for n > 0, we conclude X^J^o °^ (also notice that /(•) is bounded on Q). As 

■i=0 i=0 

for n > 0, it is clear that lim„^oo f{dn) exists. 

Let Cq be a Lipschitz constant of V/(-) on Q and an upper bound of ||V/(-)|| on the same set. Now, we 
prove lim„_^oo '^f{(^n) = 0. Suppose the opposite. Then, there exist e E (0, 00) and sequences {TOfc}fe>o, {nk}k>a 
(aU depending on ui) such that < Uk < mk+i, || V/(6'mj. )|| < e, ||V/(0„j.)|| > 2e for fc > 0, and such that 
||V/(0„)|| > e for rrik < n < Uk, k >0. Therefore, 



(65) 



e< ||V/(0„J-V/(0„.J|| <Cq||0„,-0,„J| <(72 ^ a. + 

i—mi^ 

for fc > 0. We also have 

nfc CO 
i—nii^ + l i—rrik + l 

for fc > 0. Consequently, limfc^oo X]"=nife "^^ ~ ^- However, this is not possible, since the limit process k ^ 00 
applied to ( |65] l would imply 



e< lim ||V/(0„J-V/(^?„.J|| =0. 

A; -^00 



Hence, lim„^oo V/(6'„) = 0. 



D. Convergence and Convergence Rate 

In this subsection, using the results of Subsections IIV-BI IIV-CI (Corollary |5] Lemmas |9] [TOl l. the convergence 
and convergence rate of recursion ([T]) - (|3]l are analyzed and Theorems |2] and [3] are proved. 
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Throughout the subsection, we use the following notation. For t E (0, oo), n > 0, let 

a{n, t) = max{fc > n : — Jn ^ t}. 



For < n < fc, let 



C„ = sup 

k>7i 



i—n 

i=n 
k-1 

i=n 

<fe = (v/(0„)r(4,fe+<fc), 



fe-i 



Jo 

while Sn.k = £'n,k + ^n.k 4>n,k = (t>'n^k + 4>'n,k- Then, it is straightforward to verify 

=^'« + (t/c - 7n)V/(6'„) + e„,fc, 

/(^fe) + ilk - 7n)||V/(0„)f + K,k 



(66) 
(67) 



for < 71 < fc. 

Besides the notation introduced in the previous paragraph, we also rely on the following notation in this subsection. 
For a compact set Q C O, Cq E [1,oo) denotes an upper bound of ||V/(-)|| on Q and a Lipschitz constant of 
V/(-) on the same set. A is the set of the accumulation points of {0„}„>o, while 

/ = liminf/(0„). 

n — >oc 

p and B, Q are a random quantity and random sets (respectively) defined by 

p = d{A,de)/2, B ^ [j {e' eR"^" -.we' -9\\ <mm{Se,p}} , Q ^ c\{B) 

on A, and by 

p = 0, 13 = A, Q = A 
otherwise. Overriding the definition of fi in Theorem [3] we specify random quantities S, fi, C, C as 

6 = Sqj, f^^t^Qj, C = Cqj, M^Mqj (68) 

on A, and as 

3=1, (i = 2, (7=1, M = l 
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Otherwise (Sg^a, IJ'Q,a, Mq^a are introduced in the statement of Corollary [S] later, once Theorem |2] is proved, it 
will be clear that the definitions of jl provided in Theorem [3] and in ( l68T l are equivalent). Random quantities p, q, 
f are defined in the same way as in (|5]l. Functions u{-) and v{-) are defined by 

(1/m(6'))1/p, ifu{9)>0 



u{0)=f-fi9), vi9) = 



0, 



otherwise 



for 6 ee. 

Obviously, on event A, Q is compact and satisfies A C intQ, Q C Q. Thus, jj,, M, p, q, f, v{-) are are well- 
defined on the same event (what happens with these quantities outside A does not affect the results provided in 
this subsection). On the other side. Corollary |5] implies 



im-/i<Miiv/(0)ir 



(69) 



on A for all 6* e Q satisfying 1/(9)- f\< 6. 

Lemma 12: Suppose that Assumptions [T] - 14] hold. Then, lim„^oo7^Cn = on A \ Aq (Aq is specified in the 
statement of Lemma [Toli. 



Proof: It is straightforward to verify 



for < n < fc- Therefore, 

k 



for < n < fc. Consequently, Lemma [TOl implies 



limsup7^Cn — limsupsup 

n^oo n^oc k>n 



Y "hKj 



Y 



= ln sup 



= 



on A \ A^o- ■ 
Lemma 13: Suppose that Assumptions [U - 14] hold. Let Ci — (16pM)^^ (notice that 1 < Ci < oo everywhere). 
Then, there exist a random quantity t and an integer-valued random variable a such that 0<i<l, 0<(t<oo 
everywhere and such that 



max , Ikn.fcll <(t/Ci)(7-'- + l|V/(e„)||), 

n<.k<a{n.t) 

max _ |</)„,fe| < (f/(7i)(7-"^ + ||V/(^?„)ll'), 

f{On) - f{e,^„,i)) + 2-'||V/(0„)llll^a(„,t) - ^nll < (i7(^l)7„ 



(70) 

(71) 

(72) 
(73) 



on A \ A'o for n > a. 
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Proof: Let Ci = 2C'exp(C'), C2 = ^CCi, C3 = 200^ + C2 and Ci = C2 + C3, while t = 1/(2(71 C4). 
Moreover, let 

CTi = max 



{{n>0:e„^Q]u {0}) , 
CT2 =niax {{n > : «„ > £/4} U {0}) , 



CTa = max 



({" > : 7;Cn > i7(2CiC'4)} U {0}) 

while (7 = maxjai, (72, a'3}/A\Aro- Then, it is obvious that a is well-defined, while Lemma [T2l implies < ct < 00 
everywhere. We also have 

max{(727;;cn, C37;;cn, Cst^'^c', C47;;cn, c.^i^C} < ^-'c^h, (74) 

max{C'2?, Cs?, (74*^} < 2~^C^H, (75) 

> 7a(„,t) - 7n = 7a(„,t) + l " 7n " > 3^/4 (76) 

on A \ A^o for n > a. 

Let w be an arbitrary sample from A (notice that all formulas which follow in the proof correspond to this oj). 
Since 6'„ e Q for n > cr, we have 

l|v/(0fc)|| <||v/(0„)ll + W^fiOk) - v/(0„)|| 

<\\^f{0n)\\+C\\9k-dn\\ 



<||VMOII + a,;||V/(0,;)|| + (7|K,|| 

<C(C„ + ||V/(^?„)||) + C^a,||V/(^?,)|| 
for a < n < k. Then, Bellman-Gronwall inequality yields 

||V/(0fc)ll < C(C„ + ||V/(f?„)||)exp((7(7fe - 7n)) < Cexp(C)(C„ + ||V/(0„)||) 
for a < n < k < a{n, 1). Consequently, 

fc-i 

i—n 

<Cn + Cexp(C)(C„ + \\Vf{9n)\\)hk - In) 

<^l(c„ + (7fc-7n)l|v/(^^„)||) 

for a < n < k < a{n, 1). Therefore, 

fc-i 

-i—n 

<C„ + CCi((7fc - 7„)C„ + (7fe - 7n)'l|V/(0„)||) 

<C2(Cn + (7fc-7n)'l|V/(e„)||) (77) 
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for a < n < k < a{n, 1) (notice that jk — 7ji < 1 for n < k < a{n, 1)). Thus, 

UnM\ <\\^f{On)\\\\enM\+C\\ek-Or,\\^ 

<(?2(C„I1V/(0„)|| + (7fc - 7n)'l|V/(^?„)ll') + ^CCUC + ilk - 7n)'l!V/(0„)|p) 

<C3{C + Cn||V/(^?„)|| + (7fe - 7n)'||V/(0„)||2) (73^ 

for a < n < k < a{n, 1). On the other side, combining ( |66] ), ( |67] ). we get 

/(0fc)-/(e„) =||V/(0„)llll(7fe-7n)V/(&„)|| +0„,fc 
= \\V.f{On)\\\\Ok-0n+enA+'Pn,k 

>\\^f{On)\\m-ej-\\er,,k\\)-\K,k\ 

for < n < fc. Then, (|77li, ^ yield 

<C3C + iC2 + Cs)iUVfie^)\\ + (7fc - 7n)'l|V/(0„)||2) 
<C^{C + Cn||V/(0„)|| + (7fc - 7rO'l|V/(0„)|P) (79) 

for a < n < k < a{n, 1). 

Owing to (|74li, (|75]l, (l77]l, (l78]l, we have 

\\en,k\\ <C2Cn+C2p\\^f{0n)\\ 

<CT'i(7r + l|V/(0„)||), (80) 
<2-i(7r4'(7-^^ + 7-n|V/(0„)|| + ||V/(0„)|r) 

<c•^4'(7-2'^ + ||v/(^^„)lP) (81) 

for a < n < k < a{n, t) (notice that -fk ~ < t for n < k < a{n, t)). Due to ( l67T i. ( |76] |, dSTT l. we have also 

/(^^n) - fif^ain.t)) < " (7a(„i) " 7n) 1 1 V/ (0„) | r + |0„,a(„,f)| 

< - (si/m^fiOnW + CT'i(7„'''- + l|v/(e„)|p) 

= - (3/4- cri)£||v/(^?„)f + cr47-2^'^ 

< - 2-H\\Wf{d„)f + C^Hj-^^ (82) 
for n > a (notice that Ci > 4). Consequently, 

c^H\\vf{e„)r < 2-H\\vfie„)r < c^'hn''' + (/(^a(„,t)) - /(^»)) (83) 
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for n> a. On the other side, - ^6^, ^9^, imply 

fiOn) - fidain.i)) + f iOn)\\\K^n.i) - OJ < Q (C' + C« 1 1 V/ (0„) 1 1 + ? 1 1 V/((?„ ) f ) 

<2-i(7r^t(7,;'''+7ri|V/(0„)ll + l|V/(^„)lP) 

<C^'iiln''-+\\yfiOn)r) 
<2C^'hn''' + ifi0ain,i))-fiOn)) 

for n > (7. Therefore, 

2(/(^n) - fi0ain,i))) + 1 1 V/C^n ) 1 1 1 1 e,(„,,-) - 0n\\ < ^C^'h^^^ (84) 

for n > a. Then, dTO]! - CD directly follow from (gB, dMl*- ■ 
Lemma 14: Suppose that Assumptions [T]-|4]hold. Let C2 ~ ApM^ (notice that 1 < 6*2 < 00 everywhere). Then, 
there exists an integer-valued random variable r such that {) < t < co everywhere and such that 

{<0a(n:t)) - <0n) + (^74) 1 1 V/(e„) 1 1') < 0, (85) 

{<0a(nj)) - <0n) + {1/02^0^)) Ib„ < 0, (86) 
(«(^a(n.£)) - y{0n) - t/C^) Ic„ > (87) 

on A \ A^o for n > r, where 

An ={7^K^„)I>1}U{7;J|V/(&„)||>1}, 

Bn ^{iMOn) > 1} n {A - 2}, 

Cn ={iMen) > 1} n > o} n {a < 2} 

(t is specified in the statement of Lemma [TsTl. 

Remark: Inequalities (|85T l - dSTT i can be interpreted in the following way: Relations 

{ltH0n)\ > 1 V 7,';||V/(0„)|| > 1) A n > r ^^(^.(n,*)) " «(^«) < -(<74)|| V/(0„)ll', (88) 

7^u(0„)| >lAA = 2An>r^ "(^a(r.*)) < (1 " 7C'2)ii(e„), (89) 

iMdn) >1AA<2Aji>t=^ t'(^a(„,t)) - «(^«) > i/C2 (90) 

are true on A \ A^q. 
Proof: Let 

({71 > : 0„ ^ 0} U {0}) , 

({7i>O:|«(0„)|><5}u{O}) 

and T = maxjcr, fi, f2}/A\7Vo- Then, it is obvious that r is well-defined, while Lemma [TTI implies < t < oo 
everywhere. On the other side, since r > ct on A \ Aq, Lemma [13] (inequality ( 172] )) implies 

<0ain,i)) ~ «(^n) < -{t /2)\\V f (9^.)^ + {t/C,)^-^^ (91) 



Ti = max 



T2 = max 
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on A \ A^o for n > r. As On e Q, \u{6n\ < 5 on K \ Nq for n > t, (|69]l (i.e.. Corollary |5]l yields 

\uie„)\<M\\vf{e„)\f (92) 

on A \ A^'o for n > t. 

Let Lu be an arbitrary sample from A \ A'o (notice that all formulas which follow in the proof correspond to this 
uj). First, we show (l85T l. We proceed by contradiction: Suppose that (ISST i is violated for some n > r. Consequently, 

"(^a(„,f)) - ^{(^n) + (f/4)||V/(0„)ll' > (93) 

and at least one of the following two inequalities is true: 

\u{e,,)\>ln^, I1V/(0„)|1 > 7^- (94) 

If |u(6l„)| > then ^ impUes 

||V/(a„)|p > {\u{e„)\/My^' > {l/Mf'^^~^^'^ > (4/Ci)7„-^'- 
(notice that p/fi < r, AM^/'^ < AlVP < Ci). Thus, as a result of one of (|94]i, we get 

||V/(0„)lP>(4M)7-'^ 
i.e., (f/4)||V/(0„)ll' > {i/Cih-^'. Then, ^ implies 

"(^a(„,t)) " HSn) < -(£/4)||V/(0„)|P, (95) 

which directly contradicts ( l93T l. Hence, dSST l is true for n > r. Owing to this, ( |92] | and the fact that i?„ C A„ for 
n > 0, we obtain 

(m(0,(„,,-)) - ui0„) + (i/c^Her,)) Ib„ < (w(0„(„,,-)) - uiOr.) + (Aft7c'2)||v/(0„)|p) /b„ 

< («(^?a(„,t)) - "(^n) + {i/m^f{dnW) Ib^ < 

for n > T (notice that u{dn) > on _B„; also notice that 4Af < (72). Thus, (l86T l is satisfied. 

Now, let us prove dSTl i. To do so, we again use contradiction: Suppose that dSTj i does not hold for some n > t. 
Consequently, we have fi < 2, u{d^f^,^ f-^) > and 

iMOn) > 1, (96) 

«(V,t))-«(^n)<iA. (97) 

Combining ( |96] l with (already proved) dSST l, we get ( |95l l. On the other side, ( |92] i yields 
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(notice that < u(6l„) < 5 < 1, 2//t = 1 + l/{jlf) < 1 + l/p). Therefore, ^ impHes 

t_ U{9n) -Uje^^^j-^) m(6>„) -U(6'^(„ t)) 

4- l|V/(0„)|P - 



^(H^a(„i))-«(^«)). 

Thus, t'(6'j,(„ (-)) — v{On) > tjC^i which directly contradicts Wh . Hence, ( |86] | is satisfied for n> t. 
Lemma 15: Suppose that Assumptions [T]-|4] hold. Then, 

ltu{9^) > -1, 

\\^f{9n)r<{4/i){^{u{dn))+7-^) 



(98) 
(99) 



on A \ iVo for n > r, where function i^(-) is defined by (p{x) — x I(o.oo)(2;), a; G R. 

Proof: Let ui be an arbitrary sample from A\ Aq (notice that all formulas that follow in the proof correspond to 
this uj). First, we prove ( |98] l. To do so, we use contradiction: Assume that (|98] l is not satisfied for some no > r, and 
define recursively rifc+i ~ aijii^^i) for fc > 0. Now, let us show by induction that {u{Ont,)}k>o is non-increasing: 
Suppose that u{9ni) — "(^ni_i) for < I < k and some fc > 1. Consequently, 



P 



Then, Lemma [14] (relations (l85]l, dM)) yields 

- u{9nj < -(i74)||V/(0„J|p < 0, 
i.e., u{9nk+i) < u{6n^). Thus, {M((?„^)}fe>o is non-increasing. Therefore, 

limsupM(6'„J < m(6'„J < 0. 

n — *oc 

However, this is not possible, as lim„^oo u{On) = (due to Lemma [TTli. Hence, ( |98] l indeed holds for n > r. 

Now, (|99] | is demonstrated. Again, we proceed by contradiction: Suppose that ( |99] | is violated for some n > t. 
Consequently, 

||V/(^?„)|p > (4/f)7-'^ > T-'"- 
(notice that p < fir < 2r), which, together with Lemma [14] (relations dSSl l, dSSTl), yields 

"(W-))-4^n) <-(i74)||V/(0„)|p. 

Then, ( i98] l implies 

llV/(0„)f <(4A") (u(0„) - u(0,(„.,-))) < (4/t) (^(^i(0„)) + 7„-^) . 
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However, this directly contradicts our assumption that n violates ( |99l ). Thus, ( |99] l is satisfied for n > t. ■ 
Lemma 16: Suppose that Assumptions [T]-|4] hold. Let 6*3 — 2C'|'. Then, 

liminf7Pu(6'„) < ^3 (100) 

n — >OQ 

on A \ Nq. 

Proof: We prove the lemma by contradiction: Assume that (llOOl i is violated for some sample lo from A \ A^o 
(notice that the formulas which follow in the proof correspond to this lo). Consequently, there exists tiq > t such 
that 

ltu{en) > C3 (101) 

for n < uq. 

Let {nk}k>o be defined recursively as Uk+i = a{nk,t) for fc > 0. In what follows in the proof, we consider 
separately the cases fi < 2 and // = 2. 

Case ft < 2: Owing to Lemma [14] (relations dSTl ). ( l90t ) and ( llOlb . we have 



for A: > (notice that 7^m(0„) > 1 due to dlOlb : also notice that juk+i — Ink ^ i)- Therefore, 

k-l 

viOnJ > f + (I/C2) ^(7n.+ i - 7nJ = V{9,J + (7„, - 7n„)/C'2 

1=0 

for k>0. Then, ( fToTI ) implies 

{viOno)hnk + (1 -7no/7nJ/C'2)"'' > iv{0^k) hn,)-^ = 7,i^^(e«J > 4 

for fc > 0. However, this is impossible, since the limit process fc ^ oo (applied to the previous relation) yields 
6*3 < . Hence, ( fTOOl l holds when /i < 2. 

Case ji = 2: Due to Lemma [14] (relations ([HUl, ((89ll) and (llOll i, we have 

W(^n.^j < (1 - i7(^2)u(0„J < (l - (7„. + , - 7nJ/C'2) ^(^«J 

for fc > 0. Consequently, 

k-l 



(fc-1 
-(l/C*2)X^(7n,+ i -7«.) 1 

=u(e'„J GXp (-(7„, ~ 7no)/C'2) 

for fc > 0. Then, ([TOTT i yields 

w(6'«o)7^, exp (-(7n, - 7no)/C'2) > InMOuk) > C3 



37 



for fc > 0. However, this is not possible, as the limit process k ^ oo (applied to the previous relation) implies 
C*3 < 0. Thus, (llOOl i holds in the case fi = 2, too. ■ 
Lemma 17: Suppose that Assumptions [T]-|4] hold. Let 6*4 — 6(73. Then, 



limsup7^u(6'„) < C4 (102) 



n — ^00 



on A \ iVo. 

Proof: We use contradiction to prove the lemma: Suppose that ( |102t is violated for some sample lo from A\ A'q 
(notice that the formulas which appear in the proof correspond to this uS). Since lim„^oo(7a(n i)hn) — 1, it can 
be deduced from Lemma [16] that there exist no > mo > t such that 

lfnA^--o) < 2C'3, (103) 
^iuidno) > C4, (104) 

min 7^«(^?n) > 2C3, (105) 

TTiQ <n<no 

max 7^u(6l„) < 6*4, (106) 

mo<ri<no 

and such that 

{Mmo,i)hr.rof < min{2, (1 - f/Ca)"!}. (107) 
Let Iq = a{mo,t). As a direct consequence of Lemma [T5] and ( |103t , we get 

W^fiO^jf < m (^(«(e™o)) +7™f) < 12(C3A')7™f- 
Consequently, Lemma [T3] and ( l67T i imply 

- ^^(e^o) < |</'™o,«l <(iM)(7™r + l|v/(^^™J|p) 

<ii/Cih^.T + (12(73/(71)7™!^ < 74 
for mo<n<lo (notice that j5 < 2r, f/Ci < 1/2, Ci > 24(73). Then, ( fT03] l, ( fT05] l yield 

u(0,„J > - 7™f > 2(73(7™„/7mo+i)^™f - 7™f > ((^3 - l)7™f > l^Z^ (108) 

< uie^,) + 74 < (2(73 + l)(7n/7™o)^74 < QCsln' = C^ln^ (109) 

for mp < n < Iq (notice that {'^n/lmoY — iliollmoY — 2 for mo < n < uq). Using ( |104t , ( |109t , we conclude 

^0 < no. 

In the rest of the proof, we consider separately the cases jl < 2 and jl = 2. 

Case jl < 2: Owing to Lemma [14] (relations ([87]i, ((90]l) and ( 1103b . dlOSb . we have 

«(eio) > <Qm,) + i7(^2 >(2(73)-'/^ 7mo + (7^0 - 7™o)/C'2 

>min{(2C'3)-i/^(72-i}7Zo 
=(2(73)-^/^ 7^0 
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(notice that (2(73)1/^ > Ca). Therefore, 

u{ei,) = {v{ei,))-p < 2(737,7- 

However, this directly contradicts ( I105I I and the fact that mg < Iq < tiq. Thus, (11021 ) holds when fi < 2. 
Case fi = 2; Using Lemma [14] (relations ([86]l.(l89l)) and dlOSt . we get 

However, this is impossible due to (11051 ) and the fact that mo < Iq < tiq. Hence, ( 11021 ) holds in the case fi = 2, 
too. ■ 
Lemma 18: Suppose that Assumptions [T]-|4] hold. Then, 

||e,(„.*) - ^nll < 27r^(w(e„) - uie^^^J) + 67-(^+^) (110) 

on A \ A^o for n > r. 

Proof: Let w be an arbitrary sample from A \ Nq, while n > maxjcr, t} is an arbitrary integer (notice that 
all formulas which appear in the proof correspond to these uj, n). To show (11 lOK we consider separately the cases 

||V/(0„)|| > 7n and ||V/(&„)|| < 7n 

Case ||V/(6l„)|| > ^in''^'^^^ : Due to Lemma[T3] we have 

l|v/(0„)||||0,(„,i-) - OnW < 2{u{e„) - uie.^^^i-^)) + 2{i/Cih-^\ (in) 

On the other side, since ||V/(6'„)|| > 7^'-'''''^^ > 7^'' (notice that q + I = min{(p + l)/2,r} < r). Lemma [T4l 
(relations dSSl l. dSSl l) implies 

i.e., u{9„) - u(6l^(„_£)) > 0. Then, (fTTTT i yields 

iie,(„,£) - ^?„ii <2(^.(0„) - w(^?a(„,?)))iiv/(0„)ir' + 2(£/(7i)7-'niv/(^„)ir' 

<27ri(^(0„) - uie^^„^i^)) + j-^^+(^+'^ 
<2ji+\u{e^)~u{e,^^^i))) + j-^'^+'^ 

(notice that t/Ci < 1/2; also notice that q + 1 < r, which implies 2r — {q + 1) > q + 1). Hence, (II lOl l is true 
when \\Vf{9n)\\ > 7n 

Case ||V/(6l„)|| < 7,7'''^^^- Using Lemma[T3]and we get 

k(^a(„,t)) - «(^n)l <(7a(n,t) " 7n ) 1 1 V/(^^„) 1 1 ^ + |^„,,(„,i-)| 

<£||v/(0„)|p + (tA)(7„-"^ + l|v/(0„)||2) 

<2^-2(,+i) 
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(notice that q + 1 < r < 2r and t/Ci < 1/2). On the other side, owing to Lemma [T3] and ( |66l ). we have 

\K^r.,i) - OnW <(7.(„,t) -7n)||V/(^?„)|| + |k„,,(„,,-) |! 

<£||V/(0„)ll + (i7C'i)(7„-'^ + l|V/(0„)ll) 
<2^-(«+i) 

(notice that 9 + 1 < r). Consequently, 

\Kin,t) - &n\\ <2ii+\u{e^) ~ «(0„(„,£))) + 27rV(0„) - ?/(0„(„,,-))| + 27-(«+i) 

Thus, ( fTTOl ) holds in the case ||V/(6'„)|| < 7,7''''^^^ ■ 

Lemma 19: Suppose that Assumptions [T]-|4]hold. Then, there exists a random quantity such that 1 < Cs < 00 
everywhere and such that 

limsup7«max||6'fc - 6i„|| < C5 (112) 

n— >oo k>n 

on A \ A^o- 

Proof: Let C = 9(74(g + 1) and 6*5 = 20C'i~^(l + 1/g), while a; is an arbitrary sample from A \ Aq (notice 
that all formulas which follow in the proof correspond to this lo). 
As a consequence of Lemmas [15] and [17] we get 

limsup7P|?/(0„)| < (74, (113) 

n— *oo 

limsup7P||V/(0„)|l2 < 8(74/1 (114) 

n— >oo 

Since 7^(„ - 7,, = ^ + 0(a^(„ for n ^ 00, and 

(1 - <77„)^+^ = 1 - i{q + 1)7-^ + o{^-^) 

for n 00, we conclude from dl 13t . ( II 141 ) that there exists no > max{ cr, r} (depending on uS) such that < 

2(747^^ l|V/(0„)|| < (4(74/f)7n''/', 7a(„,t) - 7« > ^2 and 

(l-t77„)«+i>l-(<7+l)7-i (115) 
for n > 7io- Then, ( l66b and Lemma [13] implv 

ll^fe-enll <(7fc-7n)llV/(0„)|l + |le,.fc|l 

<<||V/(^^„)|| + (t7(7i)(7„-'' + ||V/(0„)||) 
<8C47„-^/'+7„-'^ 

<(77-« (116) 
for no < n < A: < a(n, t) (notice that q < min{p/2, r}). 
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Let {nk}k>o be recursively defined as Uk+i = a{nk,t) for fc > 0. Due to Lemma [TSl we have 

i-i i-i i-i 

l~l I 

+ 27;J+V(0„J|+27^+V(0„J| (117) 

for ? > fc > 0. As 

7^' - 7lt\ = 7ir (l - (1 - (7n. - 7n.- J/7n.)'^') < 7^' (l - (1 - ^/7«.)'''') < + l)7l 
for z > (use ( I115l t), we get 

I oo oc 

E (7^' - 7t- < 2C^4(g + 1) E7«7+^ < ^E7n/^'''^ (118) 

i—k-\-l i—k i—k 

foT I > k>0 (notice that p ~ q > {p + 1) /2 > q + 1). Since 

(-1 

7n, = 7nfc + E(7n> + 1 " 7n J > 7n;c + (^72)(^ - fc) 
i—k 

for / > > (notice that 7^^^ £j — 7n ^ ^2) for n > no), we have 

00 00 

E7„-/^-*'^^<E(7«.+^V2)-(^+^) 

i—k i—0 

/•oo 

<7„-i«+i)+ / (7„,+W2)-(«+^)d« 
Jo 

=7-(«+i) + 2£-ir '7„7 

<{l + 2i-'q-'hn! 
for A: > 0. Consequently, dl 17l t and < ll 18l t imply 

oc 

ll^n, - II < (6 + 2C) E 7,7/^+'^ + 4C47„7+«+^ + 4C'47,7/+'^+' < 16^(1 + f^^r ')77 (119) 

i—k 

for Z > fc > (notice that p (q + 1) > (p - l)/2 > q). Using ( 1116b and ( |119l l. we get 

ll^fc — ^?»i|| <||6'fe — ^^Jijll + ll^iij — ^nJI + ll^iii — dn\\ 

<cj-^ + c^-^ + i6C(i + r^r')77 

for k > n > riQ, j > i > 1 satisfying Ui^i < n < Ui, nj-i < fc < nj. Then, it is obvious that (II 12l i is true. ■ 
Proq/ o/ r/ieoremi |2] ant/ 12} Owing to Lemmas [TTI and [T9l we have that on A \ A^o. ^ = lini„_KX) exists 
and satisfies Vf{9) = 0. Consequently, Q Q {6 e K'^* : ||6' - ^|| < Sg} on A \ A^o- Thus, random quantities p, 
q defined in this subsection coincide with p, q introduced in Theorem |3] (see the remark after Corollary |5]l. Then, 
Lemmas [HI [17] [19] imply that @ is ti'ue on A \ A^o- ■ 
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V. Proof of Propositions [T]-|4] 

Proof of Proposition Q} Owing to Conditions (i), (ii) of the proposition, for any compact set Q C 8, there 
exists a real number £q e (0, 1) such that 

eQ<r9{y\x\x)<eQ' (120) 

for all 9GQ,x,x'£X,yEy. Hence, Assumption [3] is satisfied. On the other side. Condition (ii) implies that 
re{y\x',x) has an (complex-valued) analytical continuation fjj{y\x',x) with the following properties: 

a) r,j{y\x', x) maps (rj, x, x' , y) £ C*" x X x X x y into C. 

b) fe{y\x',x) = re{y\x',x) for all 6 & Q, x,x' e X, y e y. 

c) For any compact set Q C Q, there exists a real number Sq e (0, 1) such that fri{y\x',x) is analytical in rj 
on V^^ (Q) for each x, x' e X, y e y. 

Relying on f,,(j/|x', x), we define quantities Rjj{y), (j>jj{w,y), Gn{w^y). More specifically, for rj e C^", y E y, 
Rniv) is an x Nx matrix whose entry is f,f{y\i,j), while 

\\og{e^ Rr,{y)w), ife^R^{y)w^O 
<j)r,iw,y) = i (121) 

I 0, otherwise 

, \R^{y)w/{e^R^{y)w), if e'^ R^{y)w ^ 

Gn{w,y) = < (122) 
I 0, otherwise 

for T] e C^o, yey,we C^-. 

Let Q C 6 be an arbitrary compact set. Since e^Re{y)u > N^sq for all ^^ G Q, y G 3^, w G V^^ (due to (I120l i). 
we conclude that there exists a real number 5q G (0, (5q) such that |e^i?,,(y)w| > NxeQ/2 for all rj G V5q((5), 
w G Vsq{V^''), y & y. Therefore, (l),,{w,y), G,j{w,y) are analytical in {r],w) on Vsq{Q) x Vsq{V^'') for any 
y ^ y. Consequently, \(j)ri{w,y)\, ||G,,(w, y)|| are uniformly bounded in {rj,w,y) on Vsq{Q) x VsgiV^") x y. 
Thus, Assumption |4] is satisfied, too. ■ 
Proof of Proposition^ Conditions (i), (ii) of the proposition imply that for any compact set Q C Q, there 
exists a real number eq G (0,1) such that eq < rg{y\x',x) < Eq^ for all 9 £ Q, x,x' £ X, y £ y. Thus, 
Assumption [3] holds. On the other side, as a result of Condition (ii), rg{y\x' , x) has an (complex-valued) analytical 
continuation fri{z\x',x) with the following properties: 

a) frj{z\x',x) maps {ri,x,x',z) G C^" x X x X x C^y into C. 

b) fg{y\x',x) = rg{y\x',x) for all 6 e Q, x, x' e X, y e y. 

c) For any compact set Q C 8, there exists a real number Sq G (0, 1) such that fjj{z\x',x) is analytical in 
[i], z) on (Q) X [y) for each x, x' G X. 

Relying on f^(y|a;', x), we define quantities Rr,[y), </'j)(w,2/), Grj{w,y) in the same way as in the proof of 
Proposition [T] More specifically, for 77 G C*", y G y, ^r)(j/) is an A'x x matrix whose entry is fjj{y\i,j), 
while (^^(u;, y), G^(w, y) are defined by (fTITT l. (fT22l i for 77 G C^*, y G 3^, u; G C^-. 



42 



Let Q C O be an arbitrary compact set. As N^sq < Re{y)u < N^Eq^ for any 6 £ Q, y £ y, u £ V^'^, we 
have that there exists a real number 6q £ (0, Sq) such that NxeQ/2 < le^ Rjj{y)w\ < 2NxeQ^ for all rj G Vsq {Q), 
w e ^5^(7^^"), y ey (notice that \e''^ Rjj{y)w\ is analytical in (ri,w,y) on V^^(Q) x Vi^^{V^'') x 3^). Therefore, 
(j)jj{w,y), G,-i{w,y) are analytical in (r/, w) on Vsq{Q)^Vsq{'P^'=) for any y G 3^. Moreover, \\Gn{w,y)\\ 
are uniformly bounded in {7],w,y) on V|5q((3) x V5q(7'^"=) x y. Hence, Assumption |4] holds, too. ■ 
Proof of Proposition\3\ For a £ A, P = [Pi - ■ ■ Pn^]^ ^ B, x,x' £ X, let g^{x'\x) — (ix' ,kVa{x'\'x^- Then, 
we have 

re(y|x',x) = ^ /fc(2/|a;')5e 

k=\ 

for all e 0, a;, G X, y £ y. We also have that for any compact set Q C 0, there exists a real number 
£q G (0, 1) such that eq < gg{x'\x) < Eq^ for each 6 £ Q, x, x' £ X, 1 < k < Np. Consequently, 

eq E hiy\^') < ^oivW, x) < Eq' /'^(yl^') 

fc=l fc=l 
for all 9 £ Q, X, x' £ X and any compact set Q G Q. Hence, Assumption [3] holds (set sg{y\x) — J2k=i fkiul^))- 
On the other side. Condition (i) implies that for each 1 < fc < Nfj, gg{x'\x) has an (complex-valued) analytical 
continuation g'^{x'\x) with the following properties: 

a) gri{x'\x) maps {ri,x,x') £ C^" x X x X into C. 

b) gg{x'\x) = gg{x'\x) for all e£e,x,x'£ X. 

c) For any compact set Q C Q, there exists a real number Sq £ (0, 1) such that g^j{x'\x) is analytical in 77 on 
Vg^ (Q) for each x, x' £ X. 

Relying on g^{x'\x), we define some new quantities. More specifically, for -q £ C^", w — [wi ■ ■ -WMji^ £ 'C^^, 
x,x' £ X, y £ y, let 

f^{y\x' ,x) = J2 fk{y\x')g^{x'\x), 

k=l 

x"ex 

while Rriiy) is an x matrix whose entry is f^(j/|z,j). Moreover, let (j)rj{w,y), G^(w,y) be defined 
for r] £ C**, w £ C^=", y G 3^ in the same way as in (I121l l. ( 1122b . 
Let Q C 8 be arbitrary compact set. Since 

£Q<Y1 90ix'\x)Ux < Eq' 

for all 6 £ Q, u ~ [ui ■ ■ ■umJ\'^ £ V^"', x,x' £ X, 1 < k < N/^, we deduce that there exists a real number 
Sq £ {0,~Sq) such that Re{h>^^^{x')} > eq/2, \h'^,^ix')\ < 2eq' for all ry £ Vs^iQ), w £ ^5^(7'^-), x' £ X, 
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1 < fc < iV^. Consequently, 

le'^R.ivH > \Re{e'^R^{y)w}\ = ^ ^ /,(y|a;')Re{/i^,^(a;')} > ieQ/2)^iy) > 0, 

x'ex k=i 

Nil 

max{||i?,(y)u;||,|e^i?,(yV|} < ^ ^ /,(y|x')|/i:;,^(a;')| < 2£QV(y) 

x'ex k=i 

for all rj £ Vsq{Q), w G Vsq{V^^), y £ y. Therefore, (f>jj{w,y), G,^{'w,y) are analytical in (?7,w) on Vsq{Q) x 
^5q(^^") for each y E y. Moreover, 

\\G^{w,y)\\<4eQ^ 

\M'u^,y)\ < |log|e^i?^(y)u;||+27r< |log^(y)|+log(2eQi) + 2^ 

for all f] e Vsq{Q), w e Vsq{'P'^''), y e y. Then, it is clear that Assumption |4] holds, too. ■ 
Lemma 20: Let the conditions of Proposition |4] hold. Then, (f)o{u, y), Gg{u, y) have (complex-valued) analytical 
continuations (f)ri{w,y), Grf{w,y) (respectively) with the following properties: 

i) (j)ri{w,y), G,f{w,y) map {ri,w,y) E C*" x C^'^ x y into C, (respectively). 

ii) (f)e{u,y) = (f>e{u,y), Ge{u,y) ^ Ge{u,y) for all 6* e 6, u e V^^, y E y. 

iii) For each 6 E 0, there exist real numbers Sg E (0, 1), Kg E [1, oo) such that 4>jj{w, y), Gjj{w, y) are analytical 
in (rj, w) on Vgg (6) x Vgg {V^'') for any y Ey, and such that 

\Mw,y)\<Kg{l + y^), 
\\G^{w,y)\\<Kg 

for all f] E Vs„{e), w E yEy. 
Proof: Due to Condition (i) of Proposition]?] pa{x'\x) has an (complex-valued) analytical continuation pci(x'|a;) 
with the following properties 

a) Paix'\x) maps {a,x,x') E C^" x X x X into C. 

b) pa{x'\x) — pa{x'\x) for all aE A, x,x' E X. 

c) For any a E A, there exists a real number 6a E (0, 1) such that pa{x'\x) is analytical in a on Vg (a) for 
each X, x' E X. 

On the other side, the analytical continuation q\y(vi\x) of is defined by 

(lh{y\x) = ^/\^ Gx^{~lx{y - ra:xf), 

for 6 = [/i • • • mi • • • m^J^ E C^^-, x e A", y e 3^. 

Let f^(?/|a;', x) = gb(y|a;')Pa(a;'|a;) for a e C*", 6 G C^^"', r/ = [a^ fe"^]"^, x,^' e A", y e 3^. Moreover, for 
T] E C^", yEy, Rrjiy) is an x matrix whose entry is fri{y\i,j), while ipjj{w,y), G,j{w,y) are defined 
for T] E C^", w E C^=", y G 3^ in the same way as in (1121b . (1122b . 
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Let a, (3 = [Xi ■ ■ ■ Ajv^ /ii /ijv^,]^ be arbitrarily vectors from A, B (respectively), while 9 = [a^ f]'^]'^ . Obviously, 
it can be assumed without loss of generaUty that < Ai < A2 < • • • < A^r^. Since 

Pa{x'\x)Ux > 

for all x' E X , u = [ui ■ ■ ■ wat^]^ G V^'', there exist real numbers 5i^g, eg G (0, 1) such that Rfj{y) is analytical in 
rj on Vg_^ ^ (9) for any y E y, and such that 



Re < ^ Pa{x'\x)wx > >eg, 
[xex J 



X Wj 



(123) 
(124) 



mm{R,e{li},Re{l^, - h}} > ig, 
max{|Z2;"|, \mx»\} < ig^ 

for all a G Vg^ Ja), b=[h- --In^ mi • ■■uinJ'^ € Vs,,APl w^[wi- ■■wnJ'^ G a:' G A" \ {1}, 

x" G A". Therefore, we have 



\qb{y\x)\ ^y/\l^\/7T I exp(-Re{Za;}?/^ + 2Re{lxmx}y - Re{lxml})\ 
<\/\Ix\/tt exp(-Re{;^}y^ + 2|Z^||m^||y| + |/ 

X I I '^X I j 

<(l/0i^) exp(-eey2 + 2£72|y| + £-3) 
for any b — [li ■ ■ ■ l^^ nii ■ ■ ■ rriNj'^ £ Vj^ ^ a; G A", y G 3^. We also have 

qb{y\x) 



=V\W\h\\ exp(-Re{Z^ - h}y^ + 2Re{;^ma; - hrmjy - Re{lxml - hml})\ 

<v4y7^exp(-Re{;, - h}y^ + 2(|/,||m,| + |;i||mi|)|y| + \lx\\m,f + \h\\mi\^) 
<eg^ exp{-egy^ + 4eg^\y\ + 2eg^) 

for all b = [li ■ ■ ■ In^ mi ■ ■ ■ ttin^]'^ G Vj^ ^ (/?), x £ X \ {1}, y E y. Consequently, there exists a real number 
Cg E [1,00) such that 

qb{y\x) 



< Cg, 



qb{y\^) 

\\og\qtiy\x)\\<Cgil + y^) 



for all b EVg_^^ (/?), x E X, y E y, and such that 



qb{y\x) 



<7b(2/|l) 



(125) 
(126) 

(127) 



for any b E Vg^ ^{P), x E X \ {1}, y E [-Cg^Cg]" (to show that ( 11271 ) holds for all sufficiently large \y\, 
notice that \im\y\^ocqb{y\x)/qb{y\i) = for x 7^ 1). As qi,{y\x)/qi3(y\x) is uniformly continuous in {b,y) on 
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Vj^^(/3) X [— CejCe] and lim;,^^ gb(y|a;)/g^(j/|x) = 1 for any x E X, y E y, there also exists a real number 
^2,61 e (0, 1) such that 

qb{y\x 



9/3(y|= 



- 1 

< 2 



(128) 
(129) 



for all b e Vs^ ^ (/3), x e X, y E [-Ce, Ce]. 

Let = niin{^i,9, ^2,9}, Kq = SNxCeig'^. As a result of ( 1124b . ( |125t . we have 



max{||i?^(7;)w||, |e^i?^(?/)w;|} < ^ |96(2;|a;') 



xex 



< N.,Cge-'My\l)\ 



(130) 



for all a e ^^.(a), 6 G Vs,{l3), r/ = y e 3;, w = [wi • --wnJ^ E Vs.iV^"). Using ([l23ll, (Ull, (fT27]) . 

we get 



|e^/?,(y)t.| =My\l)\ 



> 



|(7„(y|l)| Re<^ EP-(l|^)^- 



>a:eA' 



E 

x'e;t\{i} 



96(2/|2;') 



E Pa{x'\x)Wx 



>2-^ee\qb{y\l)\ 



(131) 



for all a E Vs,{a), b E 14, (/3), ry = [a^b^]^, y E [-Cg^Ce]", w = [wi ■ ■■wnJ'^ € Vs.iV'^^). Combining (fT23]) . 
(Gal, (fT28l l. (fT29l l. we obtain 



|e^i?„(y)u;| > 



E 9/3(2;|a;') ^pa{x'\x) 

'Jex xex 

> E 9/3(y|a;')Re < Ep''(^'I^)^^ f" ~ E if^(y\^' 

x'ex Vx 



E - 9/3(y|2;')) E M^'i^y 



xex 



^xex 



x'ex 



(ibiy\x') 



<ipiy\x') 



E Pa{x'\x)Wx 



xex 



x'ex 



>2~^egqp{y\l) 
>4.-'eg\qbiy\l)\ 



(132) 



for any a E Vs.ia), b E Vs,{f3), t] = [a^b^f, y E [-Ce,Ce], w = K • • • watJ'^ E Vs.iV'^^). Then, it can 
concluded from (I131l l. ( I132l l that 0,,(u', y), Gtj{w, y) are analytical in (77, w) on Vs^ (6) x V^^, {V'^") for each y G 3^. 
On the other side, (fT26] l. (fT30l l - ( fT32l i imply 

[^^(w', y)| < I log \e^R^{y)w\\ +2tt< Ce{l + j/') + \og{N^Cee,^) + 2tt< Ke{l + y'), 
||G^(ti',y)|| < 4iV.Cee72 < A'e 
for any rj E Vsf,{d), w E Vs„{'P^''), y E y. Hence, the lemma's assertion holds. ■ 
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Proof of Proposition |?} Let Q C 8 be an arbitrary compact set. Then, owing to Conditions (i), (ii) of the 
proposition, there exists a real number eg e (0,1) such that eq < Pa{x'\x) < Eq^ for all a A, x,x' £ X 
satisfying [a'^f]'^]'^ E Q for some f3 E B. Therefore, 

^QlpivW) < r0{y\x',x) < Eg'^qpiylx') 

for all a E A, [3 E B, 6 = [a'^f]'^]'^, x,x' E X, y E y satisfying 9 E Q. Thus, Assumption [3] is true. 

Since the collection of sets {V5e/2(^)}eGQ covers Q and since Q is compact, there exists a finite subset Q of 
Q such that Q is covered by {V5^/2(6')}ggg. Let Sq = min{(5e/2 : 6 E Q}, Kq — maxjXe : 9 E Q} (Sg, 
Ke are defined in the statement of Lemma l20l). Obviously, Sq E (0,1), Kq E [1,oo). It can also be deduced 
that for each 9 E Q, Vsq{9) x Vsgi'P'^-') is contained in one of the sets from the collection {Vsg{9)}g^Q. Thus, 
Vsq (Q) X Vsq {V^" ) C UggQ Vs„ (9) X Vsg {V^"). Then, as an immediate consequence of Lemma l20l we have that 
Assumption m holds. ■ 

VI. Conclusion 

We have studied the asymptotic properties of recursive maximum likeUhood estimation in hidden Markov models. 
We have analyzed the asymptotic behavior of the asymptotic log-UkeUhood function and the convergence and 
convergence rate of the recursive maximum likelihood algorithm. Using the principle of analytical continuation, we 
have shown the analyticity of the asymptotic log-likelihood for analytically parameterized hidden Markov models. 
Relying on this result and Lojasiewicz inequality, we have demonstrated the point-convergence of the recursive 
maximum likelihood algorithm, and we have derived relatively tight bounds on the convergence rate. The obtained 
results cover a relatively broad class of hidden Markov models with finite state space and continuous observations. 
They can also be extended to batch (i.e., non-recursive) maximum likelihood estimators such as those studied in [6J, 
ifTTl . Il24l . Il33l . In the future work, attention will be given to the possibility of extending the result of this paper to 
hidden Markov models with continuous state space. The possibility of obtaining similar rate of convergence results 
for non-analytically parameterized hidden Markov models will be explored, too. 
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